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Introduction 

This  document  contains  an  updated  version  of  the  Software  Development  Capability 
Evaluation  (SDCE)  model.  The  updates  incorporate  feedback  and  lessons  learned  from 
numerous  SDCE  applications  that  Aerospace  personnel  were  involved  in.  This  document 
supplements  the  Air  Force  Materiel  Command’s  (AFMC’s)  SDCE  pamphlet  [ref  I]. 


SDCE  Background 

The  SDCE  is  a  methodology  for  assessing  a  contractor’s  capabilities  in  software  and  software- 
related  systems  engineering  disciplines.  SDCE  is  based  on  the  premise  that  if  contractors  have 
defined  their  software  development  plans  and  identifed  the  software  engineering  processes,  tools, 
and  technologies  they  will  use  on  a  given  program  and  if  furthermore  they  have  past  experience 
in  the  use  of  the  identified  processes,  tools,  and  technologies,  they  present  a  lower  risk  to  the 
program  than  contractors  who  have  not.  The  methodology  was  developed  for  the  acquisition  of 
software-intensive  systems  at  AFMC  [ref  1]. 

The  SDCE  assessment  is  based  on  a  process  whereby  the  contractors  under  evaluation  provide 
information  about  their  capabilities  in  the  form  of  responses  to  SDCE  questions.  The  responses 
are  then  evaluated  using  predefined  evaluation  criteria,  and  validated  during  a  site  visit.  The 
SDCE  questions  and  associated  criteria  are  organized  into  a  structure  that  is  referred  to  as  the 
SDCE  model.  Thus,  SDCE  can  be  thought  of  as  having  two  components:  the  process  component, 
which  embodies  the  SDCE  methodology,  and  the  model  component,  which  embodies  the  SDCE 
scope.  This  document  focuses  on  the  model  component  of  SDCE. 

Although  SDCE  was  developed  using  two  predecessor  methods  as  inputs,  namely  Aeronautical 
Systems  Center’s  (ASC’s)  Software  Development  Capability/Capacity  Review  (SDCCR)  [ref  2] 
and  the  Software  Engineering  Institute’s  (SEI’s)  Capability  Maturity  Model  (CMM)  [ref  3],  it 
differs  from  most  other  software  evaluation  methods  in  the  following  three  ways.  The  first 
difference  is  that  SDCE  looks  not  only  at  software  processes  but  also  at  software  tools  and 
technologies,  primarily  in  the  System/Software  Engineering  Environment  and  the  Program 
Specific  Technologies  sections,  respectively.  The  second  difference  is  that  when  SDCE  evaluates 
the  contractor’s  ability  to  define  software  engineering  processes,  it  goes  beyond  the  existence  of 
the  processes  and  attempts  to  evaluate  their  adequacy  with  respect  to  the  program  at  hand.  For 
example,  the  responses  to  the  SDCE  questions  that  deal  with  planning  are  compared  to  related 
portions  in  the  management  section  of  the  proposal;  responses  to  the  SDCE  questions  that  deal 
with  the  choice  of  methodologies  and  tools  are  compared  to  the  technical  section  of  the  proposal 
that  describes  the  portion  of  the  system  where  these  methodologies  and  tools  will  be  used.  The 
third  distinguishing  feature  of  SDCE  is  the  tight  coupling  of  the  SDCE  process  to  the  source 
selection  rules  as  defined  in  the  Federal  Acquisition  Regulation  (FAR).  The  SDCE  model  differs 
from  models  associated  with  other  evaluation  methodologies  in  that  its  scope  is  broader  (it 
includes  tools  and  technologies),  and  its  questions  are  more  detailed. 

When  SDCE  is  applied  on  a  given  program,  the  model  is  tailored  for  that  program.  Tailoring 
of  the  SDCE  model  consists  of  selecting  a  subset  of  the  model  that  corresponds  to  the  risk  profile 
of  the  program,  while  keeping  the  size  of  the  tailored  model  consistent  with  the  resources 
available  to  perform  the  evaluation. 
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Critical 
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Capabilities 
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Program  Management 
Program  Planning  and  Tracking 
Schedules _ 


Software  schedules  are  established 
In  sufficient  detail  to  maintain 
visibility  and  control  of  the 
development  process  including 
establishment  of  any  planned 
blocks,  builds,  or  increments.  Q1 

The  program's  software  scheduling 
and  status  system  and  proposed 
schedules  are  consistent  and 
integrated  with  the  program  level 
schedules,  the  SDP,  the  IMP  and 
the  IMS.  04 


Describe  your  approach  to  establishing  the 
software  development  schedules  from  the  top 
system-level  schedule  to  the  lowest  level  detail 
schedules.  Explain  how  incremental  software 
development  schedules  are  established  C1 
Describe  the  phases,  tasks,  and  milestones  used 
in  your  lowest  level  software  schedules.  How  are 
the  durations  of  various  tasks  and  phases  of 
software  schedules  detemnined?  How  do  these 
relate  to  the  software  work  definition  packages? 
C4 

Describe  how  your  process  to  establish  software 
schedules  integrates  with  the  program’s  higher 
level  scheduling  system  (e.g.  SEMS  and  SEDS.) 

C2 


Figure  1 .  SDCE  Model  Structure 

The  SDCE  model  is  structured  into  five  layers.  The  layering  helps  the  reader  navigate  through 
the  hundreds  of  questions  included  in  the  SDCE  model,  and  facilitates  the  tailoring  of  SDCE  to 
each  program.  The  top  layer  consists  of  Functional  Areas  (FAs).  The  FAs  are  further  organized 
into  Critical  Capability  Areas  (CCAs)  which  are  groupings  of  Critical  Capabilities  (CCs).  The  two 
layers  below  the  CCs  correspond  to  the  SDCE  questions  and  the  associated  evaluation  criteria 
(Figure  1). 

The  SDCE  model  covers  those  software  and  software-related  systems  engineering  disciplines 
that  present  high  risk  for  the  success  of  a  program,  if  the  performing  contractor  has  inadequate 
capability  in  those  areas.  FAl,  Program  Management,  evaluates  the  integration  of  software 
management  with  the  overall  management  of  the  program.  FA2,  Systems  Engineering,  evaluates 
the  contrators’  software-related  system  engineering  activities.  FA3,  Software  Engineering, 
focuses  on  software  engineering  planning,  tracking,  and  the  life  cycle  functions.  FA4,  Quality 
Management  and  Product  Control,  evaluates  software  quality,  metrics,  configuration  management, 
and  documentation.  FAS,  Organizational  Resources  and  Program  Support,  evaluates  those 
capabilities  that  tend  to  be  funded  at  the  corporate  level  but  are  a  resource  to  the  program,  such  as 
training,  facilities.  Software  Engineering  Environment,  etc.  The  purpose  of  FA6,  Program  Specific 
Technologies,  is  to  assess  the  bidder’s  ability  to  use  technologies  that  are  important  to  a  given 
program. 

Our  experience  with  the  large,  unprecedented  systems  that  constitute  most  of  the  acquisitions 
for  which  SDCE  has  been  used  is  that  the  use  of  new  technologies  often  presents  high  risk  for 
these  programs.  Therefore,  assessing  the  bidder’s  capability  in  applicable  technologies  is  an 
important  element  of  the  risk  assessment  exercise  that  SDCE  implements.  A  critical  factor  in 
successful  use  of  the  SDCE  model  in  general,  and  of  FA6  in  particular,  is  to  select  evaluators  who 
are  knowledgeable  and  experienced  in  the  selected  components  of  the  SDCE  model.  This 
recommendation  is  made  in  the  SDCE  pamphlet  regarding  all  areas  of  the  SDCE  evaluation;  it  is 
particularly  applicable  to  FA6,  Program  Specific  Technologies.  Additional  information  on  SDCE 
experiences  and  lessons  learned  can  be  found  in  [ref  4]. 


Rationale  for  Updates  to  the  SDCE  Model 

There  are  three  drivers  to  the  model  updates  included  in  this  document: 
•  Population  of  FA6 
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•  Acquisition  reform 

•  Feedback  from  Space  and  Missile  Systems  Center  (SMC)  programs  on  which  SDCE  was 
used 

The  SDCE  methodology  was  defined  by  the  SDCE  Process  Action  Team  (PAT)  sponsored  by 
AFMC.  The  SDCE  PAT  completed  the  definition  of  SDCE  on  a  tight  schedule.  The  PAT 
identified  the  need  for  FA6  but  did  not  have  time  to  define  it.  FA6  was  defined  subsequently 
through  two  separate  efforts.  The  first  effort  was  undertaken  by  ASC,  and  consisted  of 
incorporating  the  technology  portions  of  the  SDCCR,  namely  the  Safety  Critical  Systems  and  the 
Complex  Hardware  Development  CCAs.  The  second  effort  was  undertaken  by  the  Aerospace 
Corporation  (Aerospace),  and  consisted  of  identifying  new  technology  areas  (Artificial 
Intelligence,  Database  Management  Systems,  Trusted  Systems,  Distributed  Systems,  and  Object 
Oriented  Developments)  and  defining  criteria  and  questions  for  them.  The  SDCE  pamphlet  was 
published  before  the  Aerospace  effort  was  completed.  As  a  result,  only  a  subset  of  FA6  was 
included  in  the  published  pamphlet  (the  Artificial  Intelligence  CCA  and  the  Database 
Management  Systems  CCA). 
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Figure  2.  Updates  to  the  June  1 994  versbn  of  the  SDCE  Model 

Acquisition  reform,  which  began  shortly  after  the  SDCE  pamphlet  was  published,  brought  with 
it  an  emphasis  on  reuse,  the  use  of  open  system  architectures,  and  a  departure  from  Government- 
mandated  process  standards.  Questions  were  added  to  help  evaluate  the  contractors’  capabilities 
in  software  reuse,  commercial  off-the-shelf  (COTS)  integration,  open  architectures,  etc.  The 
Perry  initiative,  which  came  along  with  acquisition  reform,  strongly  discouraged  acquisition 
agencies  from  mandating  DOD-defined  process  standards.  Instead  of  the  Government  defining 
the  process  that  contractors  must  follow  in  developing  software,  and  putting  that  process  on 
contract,  contractors  are  now  free  to  select  or  define  a  software  development  process  of  their 


7 


choice.  As  a  result,  several  program  offices  requested  SDCE  questions  that  can  help  them 
evaluate  the  contractor’s  ability  to  define  their  own  software  development  processes,  which  is  why 
the  Software  Development  Planning  CCA  was  rewritten. 

Other  criteria  and  questions  were  added  or  modified  based  on  feedback  from  applications  of 
SDCE  to  date.  In  particular,  questions  were  added  regarding  the  contractor’s  ability  to  correct 
the  Year  2000  problem,  and  the  whole  section  on  Configuration  Management  (CCA  4.7)  was 
rewritten  to  facilitate  tailoring.  The  differences  between  the  SDCE  model  provided  in  this 
document  and  the  SDCE  model  published  in  the  AFMC  pamphlet  are  illustrated  in  Figure  2,  and 
summarized  in  Table  1. 


Aerospace’s  Contributions 

Aerospace  has  been  involved  with  SDCE  since  its  inception.  When  the  AFMC  PAT  was  formed, 
it  was  partly  in  response  to  a  request  from  the  Commander,  SMC  for  AFMC  to  identify  a  software 
evaluation  method  suitable  for  SMC.  The  SMC  commander’s  request  was  preceded  by  an 
Aerospace  report  that  analyzed  the  existing  contractor  evaluation  methodologies,  and 
recommended  development  of  a  new  method  that  should  take  the  best  from  SETs  CMM  and 
ASC’s  SDCCR  [ref  5].  As  a  result.  Aerospace  was  invited  to  participate  in  the  SDCE  PAT,  and 
made  significant  contributions  to  the  initial  version  of  the  SDCE  model.  Subsequently, 

Aerospace  supported  all  SMC  acquisitions  that  used  SDCE,  and  provided  the  majority  of  the 
evaluators  for  these  SDCEs. 

In  addition  to  the  SDCE  activities  funded  directly  by  programs.  Aerospace  undertook  two 
internally  funded  efforts  aimed  at  providing  cross-program  SDCE  support  and  enhancing  the 
method.  The  first  of  these  efforts,  the  SDCE  Applications  Engineering  Method  (SDCE 
Applications  EM),  aimed  at  promoting  the  use  of  SDCE  and  developing  support  material  for 
applying  it.  Much  of  FA6  [ref  6]  was  developed  under  this  effort.  The  second  effort,  the  SDCE 
Metrics  Mission  Oriented  Investigation  and  Experimentation  (SDCE  Metrics  MOIE),  aimed  at 
collecting  and  analyzing  data  from  SDCE  applications  at  SMC,  and  using  results  of  the  analysis  to 
develop  improvements  to  the  method  and  guidelines  for  its  use  [ref  7].  The  Object  Oriented 
Development  portion  of  FA6,  along  with  the  updates  to  FAl  through  FAS  were  outputs  from  the 
SDCE  Metrics  MOIE. 

This  document  is  intended  for  use  in  conjunction  with  the  AFMC  pamphlet,  as  a  replacement 
to  the  SDCE  model  charts  in  Chapter  3  “Description  of  the  SDCE  Model”  and  an  update  to 
Chapter  5  “Model  Criteria  and  Questions.” 


Table  1.  Traceability  Between  1994  and  1998  SDCE  Models 


cc 

Criteria/ 

Questions 

Change  Description 

2.2.1 

C5,Q5 

Ad(jition  (Architecture) 

2.6.1 

C5,Q5 

Addition  (Year  2000) 

2.72 

C3-5,Q4-6 

Addition  (Reuse) 

3.1.3 

Section  title 

C1-2,  Q1-2 

C3.Q3 

Modification  (rewrite  of  Software  Development  Planning  CCA) 

Modification  (rewrite  of  Software  Development  Planning  CCA) 

Deletion  (rewrite  of  Software  Development  Planning  CCA) 

3.1.4 

C1-2,  Q1-3 
C3-8,Q4-8 

Modification  (rewrite  of  Software  Development  Planning  CCA) 

Deletion  (rewrite  of  Software  Development  Planning  CCA) 

3.4.1 

C8,  Q11 

Addition  (Architecture) 

4.7.1 

C1-2,  Q1-2 
C3^,Q3-6 

Modification  (rewrite  of  Software  Configuration  Management  CC) 

Deletion  (rewrite  of  Software  Configuration  Management  CC) 
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Table  1.  Traceability  Between  1994  and  1998  SOCE  Models  (Cont'd) 


cc 

Criteria/ 

Questions 

Change  Description 

472 

C1-3,  Q1-<3 

Modification  (rewrite  of  Software  Configuration  Management  CC) 

C4-8!Q4-9 

Deletion  (rewrite  of  Software  Configuration  Management  CC) 

C1,Q1 

Modification  (rewrite  of  Software  Configuration  Management  CC) 

us^m 

C2.  Q2-3 

Deletion  (rewrite  of  Software  Configuration  Management  CC) 

47.4 

C1->3.  Q1-3 

Modification  (rewrite  of  Software  Configuration  Management  CC) 

C4-6,Q4-6 

Deletion  (rewrite  of  Software  Configuration  Management  CC) 

47.5 

C1-2,  Q1-2 

Modification  (rewrite  of  Software  Configuration  Management  CC) 

C3^.Q3-4 

Deletion  (rewrite  of  Software  Configuration  Management  CC) 

572 

C2,Q4 

Modification  of  software  tools  question  (clarification) 

622 

Q8 

The  text  of  this  question  in  the  Safety  Critical  Systems  CCA  had  been  inadvertently  left  out  of 

Safety 

the  AFMC  pamphlet 

6.3.4 

All 

Additions  ofVHDLCC 

Complex 

Hardware 

6.42 

All 

The  old  CC  6.4.2  (Database  Tools)  was  separated  Into  two  CCs:  6.4.2  (Database  Design) 

DBMS 

and  6.4.3  (Database  Tools),  and  both  were  expanded 

Q6 

Modification  (clarification) 

Q7 

Deletion  (subsumed  by  Q6) 

6.4.3 

All 

The  old  CC  6.4.2  (Database  Tools)  was  separated  into  two  CCs:  6.4.2  (Database  Design) 

and  6.4.3  (Database  Tools),  and  both  were  expanded 

DBMS 

C1 

Modification  (references  to  06-1 1  added) 

6.4.4 

Q3-4 

Modification  (CC  rewrite) 

DBMS 

Q5 

Deletion 

Q6 

Modification  (becomes  05) 

6.4.5 

Q6~7 

Additional  questions  for  the  Database  Duality  Assurance  CC 

DBMS 

6.4.6 

C1 

Modification  (references  to  06-7  added) 

DBMS 

Q5,Q9 

Modification  (CC  rewrite) 

6.5.1-6.5.3 

All 

New  CCA 

Trusted 

Systems 

6.6.1-6.6.4 

All 

New  CCA 

Distributed 

Network 

Systems 

6.7.1-6.7.5 

All 

New  CCA 

OO  Devel. 

9 


Revised  SDCE  Model 
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Functional  Area  1:  Program  Management 


SDCE  Model 


Figure  3.  Program  Management 


.1 

.1.1 


Program  Management 

Management  Authority,  Responsibility,  and  Accountability 
Organizational  Approach 


1 

1 

1 


Cl  The  software  development  and 

management  functions  are  organized 
consistent  with  the  proposed  overall 
system  development  organizational 
structure  (e.g.,  straight  functional, 
Integrated  Product  Teams  (IPTs))  and 
include  identified  support  functions  to 
the  system  engineering,  subcontractor 
development  and  other  functional 
development  support  activities  as  needed. 
Q1  Q3 


C2  The  program  organization  includes 

specific  elements  composed  of  systems, 
software  and  hardware  engineering  that 
are  responsible  for  the  allocation  of 
requirements  and  derivation  of  design 
from  the  System  Specification  to  the 
Software  and  Hardware  Configuration 
Item  (HWCI)  specifications.  Q2 


C3  The  software  engineering  organization  is 
structured  such  that  all  program  software 
(including  support  software) 
development  is  assigned  to  specific 
organizational  elements.  Q4 


C4  The  total  software  development 
organization  is  defined  and 
responsibilities  assigned,  including 
identified  elements  responsible  for  the 
management  and  control  of 
subcontractor-developed  and  vendor- 
delivered  software.  Q3 

C5  The  organizational  structure  integrates 
special  technology-driven  resource 
requirements  (e.g.,  specialists  in 
languages,  architectures,  methods,  tools 
not  assigned  full  time  to  the  program) 
into  the  program  organizational  working 
structure.  Q5 


Q1  Describe  the  total  software  development 
organization,  top  to  bottom,  including 
intermediate  organizational  supervisory 
levels.  How  is  this  software  development 
function  organizationally  integrated  and 
consistent  with  the  program’s  overall  system 
development  organizational  structure  (e.g., 
straight  functional,  IPTs,  etc.)?  Describe  the 
major  software  subcontractors’ 
organizations  to  develop  software.  Describe 
any  formal  agreements  between  team 
members  that  define  specific  responsibilities 
for  development.  Cl 

Q2  Describe  the  organizational  elements 
responsible  for  the  allocation  of 
requirements  and  derivation  of  design  from 
the  System  Specification  through  the 
various  levels  of  design  and  requirements  to 
the  Software  Requirements  Specifications, 
HWCI  specifications.  Interface  Requirements 
Specification,  and  top-level  design.  Include 
teaming  and/or  subcontracting  elements. 

C2 

Q3  How  does  this  structure  provide  the 

necessary  support  functions  to  related 
development  functions  such  as  systems 
engineering,  subcontractor  development 
management,  vendor  management  and  other 
functional  development  support  activities? 
Cl  C4 

Q4  Describe,  within  the  identified  software 

development  organization  and  structure,  the 
responsibility  assignments  for  all  program 
software  including  support,  integration,  and 
test  software.  C3 


Q5  How  is  the  organization  structured  to  meet 
the  program  needs  for  specialized 
technology  skills  that  are  driven  by  program 
requirements  such  as  language  and 
architecture  specialists  who  may  not  be 
required  full  time  on  the  program?  C5 
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1  Program  Management 

1.1  Management  Authority,  Responsibility,  and  Accountability 

1.1.2  Management  Control 


Cl  The  chief  engineer/system  engineer,  or 
equivalent,  is  organizationally  responsible 
for  all  technical  activities  on  the  program. 

Q1  Q2 


C2  Responsibility  for  control  of  all  software 
development  resides  within  the  program 
organization,  including  subcontracted, 
simulation,  integration,  and  test  software. 

Q3  Q4  Q5  Q6 


Q1  Where  does  the  overall  technical 

responsibility  for  the  program  development 
reside?  Identify  program  technical  activities 
that  do  not  report  through  the  lead  technical 
manager.  To  whom  does  the  lead  technical 
manager  (Chief  Engineer)  report?  C 1 

Q2  Where  does  the  program  management 
responsibility  reside?  Identify  any 
development  activity  outside  the  control  of 
the  program  manager,  upon  which  the 
program  is  dependent;  e.g.,  reusable 
software,  tools  to  be  acquired,  components 
being  developed  by  another  organization, 
etc.  How  does  the  program  manager 
influence  and  status  these  components?  Cl 

Q3  Where  does  the  overall  software 

responsibility  reside?  Does  all  software 
development  report  through  a  single 
software  manager?  Is  all  software,  including 
that  developed  by  teaming  associates  and 
subcontractors,  support,  simulation, 
integration  and  test  software,  included  in  the 
overall  software  management  responsibility? 
If  not,  how  is  this  other  software  managed, 
interfaced  and  integrated  with  the  central 
software  effort?  C2 


Q4  How  is  software  managed  within  the 

Integrated  Product  Development  Team 
approach  to  the  system  development?  How 
is  software  managed/developed  within 
subsystems  and  across  interrelated 
subsystems?  C2 

Q5  Who  is  responsible  to  see  that  the  Software 
Development  Plans  and  Software 
Development  Standards  are  followed?  C2 

Q6  Describe  the  specific  management  functions 
applied  to  control  software  development. 

C2 
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1 

1.2 

1.2.1 


Program  Management 
Program  Planning  and  Tracking 
Planning 


Cl  The  program  planning  accounts  for  the 
integration  of  software  development  and 
management  with  system  and  hardware 
management.  Ql 

C2  The  proposed  program  planning 
approach  includes  planning  for 
personnel  qualifications,  quantities,  skill 
types,  need  dates,  and  required  training. 

Q7 


C3  The  program  planning  includes  the 

necessary  reviews,  accountability,  status 
assessment,  schedule  control  and 
reporting  to  manage  the  software  related 
system  development  activities  leading  to 
the  definition  of  the  software 
requirements  baseline.  Q2  Q3 


C4  The  program  planning  includes  a  series 
of  technical  and  management  reviews 
with  associated  completion  criteria  that  is 
used  to  control  the  development 
progress.  Q3 


C5  Specific  management  planning  processes 
are  defined  to  account  for  the  proposed 
software  development  methodologies  and 
implementation  language  selected.  Q4 


C6  Variance  thresholds  are  established  for 
critical  status  metrics  (e.g.,  size,  cost 
effort,  progress,  and  schedule).  Q5  Q6 


Ql  How  is  your  software  development  planning 
integrated  with  systems  management  and 
hardware  management?  Cl 

Q2  Describe  your  planning  process  used  to 
establish  the  front-end  software  related 
system  development  activities.  Describe 
your  process  to  status  and  report  these 
activities  including  specific  criteria  and 
control  measures.  Who  is  responsible  to 
perform  these  front-end  management 
activities?  C3 

Q3  Describe  your  technical  and  management 
reviews  used  to  control  the  development 
progress  throughout  the  entire  development 
period.  Define  these  events  and 
corresponding  criteria.  How  are  these  events 
incorporated  into  the  Systems  Engineering 
Master  Plan,  Systems  Engineering  Master 
Schedule,  Systems  Engineering  Detailed 
Schedule,  and  the  Software  Development 
Plan?  C3  C4 

Q4  Describe  any  special  management  planning 
processes  used  as  a  result  of  the  selected 
software  development  methodologies  and 
programming  languages.  How  do  these 
processes  vary  from  your  standard  software 
development  management  activities?  C5 

Q5  Identify  the  software  tracking  metrics  to  be 
used  on  this  program.  Describe  your 
process  for  monitoring  critical  status  metrics 
or  indicators.  How  do  you  determine  when 
management  action  is  required?  Describe 
the  conditions  that  would  result  in 
management  action  for  each  established 
metric  or  indicator.  C6 

Q6  Where  is  this  metrics  monitoring  process 
documented?  C6 


Q7  Describe  how  your  program  planning 
includes  provisions  for  personnel 
qualifications,  quantities,  skill  types,  need 
dates,  and  required  training.  C2 


1 

1.2 

1.2.2 


Program  Management 

Program  Planning  and  Tracking 

Contract  Work  Breakdown  Structure  (CWBS) 


Cl  The  proposed  CWBS  and  internal  CWBS 
generation  procedures  identify  software 
elements  to  levels  that  support  software 
management  visibility  and  are  compatible 
with  cost  reporting  and  program  RFP 
requirements.  Q1  Q2  Q3  Q4 

C2  The  proposed  CWBS  and  internal  CWBS 
generation  procedures  identify  how  the 
CWBS  links  with  and  traces  to  the  work 
definition  system  down  to  and  including 
the  software  work  packages.  Q5 

C3  The  program  has  a  mutually  consistent 
and  integrated  CWBS,  work  definition, 
scheduling,  and  cost  tracking  system  and 
is  used  as  the  basis  for  program  status 
and  control.  Q6 


Q1  How  and  at  what  level  is  software  structured 
in  the  CWBS?  Provide  examples  of  your 
recent  CWBSs  that  include  major  software 
development.  Cl 


Q2  What  are  the  factors  and  criteria  for 
determining  software  level  within  the 
CWBS?  Cl 


Q3  Identify  your  internal  standards  (criteria) 
for  identifying  software  work  within  the 
CWBS.  Cl 


Q4  At  what  level  (of  CWBS)  is  software  reported 
in  the  cost  performance  report?  Cl 

Q5  Describe  the  overall  flow  of  work  definition 
from  the  CWBS  down  through  detailed  work 
definition  including  cost  accounts  and  work 
packages.  Explain  (illustrate)  how  the 
software  work  package  interfeces  with  and  is 
traceable  to  the  software  elements  of  the 
CWBS.  C2 

Q6  Describe  how  your  CWBS  procedures 

integrate  with  your  work  definition  process, 
scheduling  process,  and  cost  tracking 
system.  Describe  how  the  CWBS  is  used  to 
support  program  status  and  control.  C3 


1 

1.2 

1.2.3 


Program  Management 
Program  Planning  and  Tracking 
Work  Packages 


Cl  A  documented  process  exists  for  defining 
software  development  work  packages, 
including  schedules  and  manpower 
allocations.  This  process  includes  rules 
and  criteria  for  formulating  software 
work  packages.  Q1  Q2  Q3 

C2  The  software  work  package  is  used  to 
manage  the  work  and  is  used  as  the  basis 
for  cost  performance  reporting.  The  cost 
performance  reporting  system  includes 
all  of  the  software  development  tasks  and 
activities.  Q4 

C3  The  software  work  packages  include 
planned  and  actual  effort  expenditures. 

Ql 


C4  The  software  work  packages  have 

completion  milestones,  with  associated 
criteria  scheduled  consistent  with  the 
program  requirements  (e.g.,  cost 
performance  reporting  (CPR)  and 
Cost/Schedule  Control  System  Criteria 
(CSCSC)).  Q1Q5 


C5  The  scheduling  information  contained  in 
the  software  work  packages  is  consistent 
with  the  program  scheduling  system.  Q3 

Q5 


C6  The  CWBS  and  Work  Package  Definition 
System  provides^  a  correlation  to  the 
software  structure.  Q6 


Ql  Explain  your  method  for  defining  software 
work  package  including  schedules  and 
manpower  allocations.  Identify  rules, 
essential  elements  and  criteria  for  an 
acceptable  software  work  package  within 
your  system.  Cl  C3  C4 

Q2  Identify  the  document  that  fully  describes 
the  software  work  package  method.  Cl 


Q3  Describe  how  your  cost  account  and  work 
package  definition  process  is  integrated  with 
the  system  and  software  scheduling  system. 
Explain  how  the  software  work  package  is 
used  to  plan  the  work,  i.e.,  plan,  define  and 
assign  resources  (manpower  loading)  and 
responsibility.  How  are  the  Contract  Work 
Breakdown  Structure  (CWBS),  cost  accounts 
and  work  packages  used  to  status  and  report 
progress  of  the  program  in  terms  of  effort 
(cost)  and  schedule?  Cl  C5 

Q4  Explain  how  the  software  work  package  is 
used  in  CPR.  Explain  how  Earned  Value 
data  from  software  development  activities  is 
included  in  the  CPR.  How  do  these  differ 
for  varying  elements  of  the  CWBS?  How  is 
the  CPR  used  to  establish  a  true  indication 
of  software  cost  considering  the  effects  of 
general  or  overhead  functions  such  as 
software  management,  configuration 
management,  software  quality  assurance, 
internal  independent  verification  and 
validation,  etc.?  C2 

Q5  How  do  the  Work  Package  milestones  and 
completion  criteria  correlate  with  Systems 
Engineering  Master  Plan/Systems 
Engineering  Master  Schedule 
(SEMP/SEMS)  milestones?  Is  the 
completion  criteria  and  earned  value  within 
the  CPR  or  CSCSC  system  related  to  the 
SEMP/SEMS  completion  criteria?  C4  C5 

Q6  Explain  how  the  CWBS,  cost  accounts  and 
work  packages  correlate  with  the  software 
structure.  C6 
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1 

1.2 

1.2.4 


Program  Management 
Program  Planning  and  Tracking 
Schedules 


Cl  Software  schedules  are  established  in 

sufficient  detail  to  maintain  visibility  and 
control  of  the  development  process 
including  establishment  of  any  planned 
blocks  builds  or  increments.  Q1  Q5 

C2  The  program’s  software  scheduling  and 
status  system  and  proposed  schedules  are 
consistent  and  integrated  with  the 
Software  Development  Plan  (SDP)  and 
the  program  system  level  schedules, 
including  the  Systems  Engineering 
Master  Plan,  Systems  Engineering  Master 
Schedule,  and  Systems  Engineering 
Detailed  Schedule  (SEMP/SEMS/SEDS) 
as  appropriate.  Q3  Q4  Q6 

C3  The  proposed  schedule  duration  for 
software  development,  software 
integration,  and  software/hardware 
integration  is  consistent  with  the  effort  to 
be  accomplished  as  estimated  with 
established  estimating  models  and  the 
offeror’s  historical  data.  Q8 

C4  The  lowest  level  software  schedules 
include  task,  phase,  and  milestone 
definitions  that  are  consistent  with  the 
software  work  definition  packages.  Q2 


C5  A  process  is  defined  to  maintain 

consistent  software  schedule  information 
across  various  disciplines  including 
engineering,  management,  and  the  Cost 
Performance  Reporting  System.  Q7 


Ql  Describe  your  approach  to  establishing  the 
software  development  schedules  from  the 
top  system  level  schedule  to  the  lowest  level 
detail  schedules.  Explain  how  incremental 
(block,  build)  software  development 
schedules  are  established.  Cl 

Q2  Define  the  phases,  tasks,  and  milestones  used 
in  your  most  detailed  (lowest  level)  software 
schedules.  How  are  the  duration  of  various 
tasks  and  phases  of  software  schedules 
determined?  How  do  these  relate  to  the 
software  work  definition  packages?  C4 


Q3  Describe  how  your  process  to  establish 
software  schedules  integrates  with  the 
program’s  higher  level  scheduling  system 
(e.g.,  SEMS  and  SEDS).  C2 


Q4  Describe  your  approach  to  defining 

milestones  with  completion  criteria  in  the 
SEMS  and  SEMP.  How  is  this  approach 
correlated  with  the  software  scheduling 
system  including  use  of  software  milestones 
and  criteria?  C2 

Q5  Describe  your  method  for  monitoring  and 
statusing  software  development  schedules. 
Who  is  responsible  for  this  function?  How 
is  each  level  of  schedule  addressing  software 
used  in  the  management  process?  Which 
level  is  used  as  the  baseline  to  track  and 
report  status?  Cl 

Q6  Describe  how  software  schedules  are 

defined,  referenced,  used,  and  updated  in 
the  SDP.  C2 

Q7  Describe  your  approach  to  maintaining 
consistent  software  schedule  information 
across  various  disciplines  (e.g.,  management, 
engineering,  and  program  control  (Cost 
Performance  Reporting  and  Cost/Schedule 
Control  System  Criteria)).  C5 
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1 

1.2 

1.2.4 


Program  Management 
Program  Planning  and  Tracking 
Schedules  (Cont’d) 


Q8  Describe  this  program’s  software 

development,  software  integration,  and 
software/hardware  integration  schedules 
including  time  phasing  and  duration.  Also 
describe  scheduling  of  any  planned 
increments  or  blocks.  How  were  these 
schedules  derived?  Relate  proposed 
schedules  to  the  effort  to  be  accomplished 
(man  months),  available  personnel  resources 
and  your  past  schedule  accomplishments  on 
similar  programs.  C3 
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.3 

.3.1 


Program  Management 
Subcontractor  Management 
Capability  Evaluation 


f 


Cl  As  part  of  the  subcontractor  selection 

process,  documented  procedures  exist  to 
evaluate  subcontractors’  capability  and 
capacity  to  develop  software.  Q1  Q2 


Ql  How  are  potential  subcontractors’  software 
development  capabilities  and  capacities 
evaluated  prior  to  selecting  a  specific 
subcontractor?  Cl 

Q2  Where  is  this  procedure  for  evaluating 

subcontractors’  software  capabilities  and 
capacities  documented?  Cl 
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1 

1.3 

1.3.2 


Program  Management 
Subcontractor  Management 
Subcontractor  Development  Management 


Cl  The  proposed  subcontractor  management 
process  is  integral  to  the  system  program 
management  process  and  provides 
integrated  reporting  and  control  of  the 
subcontractor  software  development 
activities  consistent  with  the  program’s 
management  control  system.  Q1 


C2  The  system-level  engineering 
management  controls  including 
SEMP/SEMS/SEDS  are  levied  on 
subcontractors.  Q2 

C3  Periodic  management  and  technical 
reviews  to  address  subcontractor 
development  progress  are  conducted  and 
are  reflected  in  the  program’s 
SDP/SEMP/SEMS/SEDS.  Q1 

C4  A  process  is  defined  to  specify  and 

control  the  subcontractor’s  performance 
requirements,  interfaces,  deliverables  and 
product  testing.  Q3 

C5  A  documented  process  exists  which 
requires  reviewing  and  assessing  the 
technical  content  of  subcontractor 
generated  design  information  and 
documentation.  Q4  Q5  Q6  Q7  Q8 


C6  The  software  test  and  verification  process 
includes  subcontractor  developed 
software  and  incorporates  the 
subcontractor  software  test  and 
verification  management  and  results  into 
the  overall  hierarchical  test  process.  Q9 

C7  The  subcontractor’s  defined  software 
cost  status  and  reporting  system  is 
compatible  with  the  program  cost  status 
and  reporting  requirements.  QIO  Qll 


Q1  Fully  describe  your  process  for 

subcontractor  management  including 
reporting  and  control  of  the  subcontractor 
software  development  activities.  How  does 
this  process  relate  to  and  integrate  with  your 
overall  system  program  management 
approach?  Describe  how  the  subcontractor 
management  and  review  activities  are 
reflected  in  the  program  level  Systems 
Engineering  Master  Plan,  Systems 
Engineering  Master  Schedule,  and  Systems 
Engineering  Detailed  Schedule 
(SEMP/SEMS/SEDS).  Cl  C3 

Q2  Using  this  approach,  how  are  the  SEMP, 
SEDS,  SEMS,  and  the  Software 
Development  Plan  (SDP)  flowed  down  to 
the  subcontractors?  C2 

Q3  How  do  you  specify  and  control  the 
subcontracted  software 
technical/performance  requirements, 
interfaces,  deliverables,  and  product  testing 
(test  requirements  and  criteria)?  C4 

Q4  Describe  your  process  for  establishing  and 
conducting  periodic  management  and 
technical  reviews  and  interchanges  with  your 
subcontractors.  C5 

Q5  Describe  your  process  to  integrate 

subcontractor  design  information  and 
documentation  into  the  system 
documentation.  Identify  the  technical 
products  and  the  set  of  software 
documentation  you  require  as  deliverables 
from  your  subcontractors.  C5 

Q6  What  role  does  software  documentation  play 
in  statusing  the  subcontractor’s 
development  activities?  Describe  how 
documentation  is  reflected  in  the 
SEMP/SEMS  events  and  criteria.  C5 


Q7  How  is  this  information  reviewed  and 
evaluated  for  adequacy?  What  are  the 
criteria  for  complete  documentation 
regarding  both  individual  documents  and 
the  set  as  a  whole?  C5 
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1 

1.3 

1.3.2 


Program  Management 
Subcontractor  Management 

Subcontractor  Development  Management  (Cont’d) 


C8 


The  software  size  control  program  Q8 

established  for  the  program  is  applied  to 
the  subcontractor  effort  and  monitored 
throughout  the  development.  Q12 

Q9 


Who  within  your  organization  is  responsible 
to  review  and  approve  subcontractor 
software  documentation?  C5 


What  technical  completion  criteria  for 
software  are  identified  in  the  subcontract? 
Describe  your  test  criteria  and  procedures 
for  accepting  subcontracted  software.  How 
is  subcontracted  software  incorporated  into 
your  software  integration  and  test  process? 
C6 


QIO  How  is  management  visibility  into  the 
subcontracted  development  efforts 
established  and  maintained?  Specifically, 
how  do  you  assess  a  subcontractor’s 
software  development  status?  What  is  the 
basis  for  this  assessment?  How  are  Work 
Packages  used  in  this  assessment?  What 
metrics/indicators  are  required  from  the 
subcontractor?  How  often  is  this 
information  submitted  and  in  what  format? 
C7 


Qll  What  formal  reporting  do  you  require  of 
your  subcontractor  relative  to  software? 
How  is  this  reporting  tied  to  the 
subcontractor’s  Contract  Work  Breakdown 
Structure  (CWBS)  and  in  turn  to  your 
CWBS?  To  what  level  do  you  require  the 
subcontractor’s  software  to  be  identified 
and  reported  in  his  cost  performance 
report?  C7 

Q12  How  is  software  size  control  established  and 
applied  to  subcontracted  software 
development?  C8 
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1 

1.3 

1.3.3 


Program  Management 
Subcontractor  Management 
Subcontractor  Planning 


Cl  The  software  development  planning 

process  includes  subcontracted  software 
development.  Q1 


C2  Program-level  SDP  requirements  and 
process  requirements  are  levied  on 
subcontractors.  Subcontractor  SDPs  are 
consistent  with  the  prime  contractor’s 
software  development  planning.  Q2 


C3  Subcontractor  SDPs  are  reviewed  and 
approved  by  the  prime  contractor.  Q3 


C4  Procedures  ensure  that  the  program’s 
development  standards  and  procedures 
are  applied  to  subcontractor  development 
efforts  or  a  process  is  in  place  to  ensure 
that  subcontractor  standards  and 
procedures  are  used  which  are 
compatible  with  the  program’s 
development  processes.  Q3  Q4 

C5  The  program’s  software  documentation 
requirements  and  documentation 
approach  are  levied  on  subcontractors 
developing  software.  Q2 


C6  If  award  fees  or  incentives  are  established 
for  subcontractor-developed  software, 
measurable  award  fee  or  incentive  criteria 
are  established.  Q5 


Q1  How  does  your  Software  Development  Plan 
(SDP)  incorporate  the  planning  for 
subcontracted  software  development? 
Describe  specific  planning  coverage  areas 
required  to  manage  subcontracted  software 
development.  Is  this  planning  based  on  a 
written  organizational  policy  for  managing 
software  subcontractors?  Cl 

Q2  Describe  how  the  program’s  SDP  and 
process  requirements  including 
documentation  requirements  are  flowed 
down  to  subcontractors  developing  software. 
How  do  you  ensure  that  subcontractor  SDPs 
are  consistent  with  your  software 
development  planning?  C2  C5 

Q3  Are  subcontractor  SDPs  reviewed  and 
approved?  How  are  these  plans 
incorporated  into  your  subcontractor 
development  monitoring  and  tracking 
activity?  C3  C4 

Q4  Do  you  apply  your  software  standards  to 
your  subcontractor(s)?  If  not,  what 
standards  are  required?  How  do  you  ensure 
that  standards  are  followed  and  are 
compatible  with  the  program’s  development 
processes?  C4 


Q5  Describe  your  approach  to  establishing 

award  fees  and  incentives  for  subcontractor- 
developed  software.  Are  predefined  criteria 
established?  Describe  the  nature  of  these 
criteria.  Do  you  plan  to  use  award  fees  or 
incentives  on  this  contract?  C6 
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1 

1.3 

1.3.4 


Program  Management 
Subcontractor  Management 
Subcontractor  Configuration  Management 


Cl  The  subcontractors’  software 

configuration  management  system  is 
documented  and  is  compatible  and 
consistent  with  and  supports  the  overall 
program  software  configuration 
management  needs.  Q1 

C2  The  subcontractors’  software 

configuration  management  system  is 
reviewed  and  verified  by  the  prime 
contractor  to  be  totally  compliant  with 
program  requirements  and  needs  early  in 
the  development  phase  (i.e.,  prior  to 
Preliminary  Design  Review).  Q2 


Qi  Describe  how  your  subcontractor  software 
configuration  management  system 
integrates  with  the  overall  program  software 
configuration  management  system.  Describe 
how  they  are  mutually  compatible  and 
consistent.  Cl 

Q2  How  do  you  review  your  subcontractors’ 

internal  software  configuration  management 
system  compliance  to  the  program-level 
configuration  management  requirements? 
How  early  in  the  development  is  this 
compliance  verified?  C2 
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1 

1.4 

1.4.1 


Program  Management 
Legal  and  Contracting  Issues 
Software  Rights 


Cl  The  software  rights  required  for  follow- 
on  support  and  system  maintenance 
including  development  System/Software 
Engineering  Environments  (S/SEEs)  and 
tools  are  identified,  including 
subcontracted  and  vendor  software.  Q1 
Q2 


C2  The  identified  restrictions  on  the  use  of 
the  delivered  software  are  compatible 
with  the  intended  system  operation  and 
support  concept.  Q1  Q2 


C3  The  software  development  process  is 
compatible  with  the  limitations  imposed 
by  restricted  rights  and  licensing 
restrictions  for  software.  Q2 


C4  The  identified  rights  to  reused  software, 
either  previously  developed  or 
concurrently  developed  across 
development  teams,  including 
subcontracted  and  vendor  software  on 
this  program,  are  consistent  with  the 
program’s  life  cycle  support  and 
maintenance  needs.  Qi 

C5  The  contractual  terms  proposed  for  use 
in  acquiring  subcontracted  and  vendor 
software  includes  provisions  for  the 
necessary  rights  to  software  and  are 
compatible  with  the  prime  contractor’s 
contractual  obligations  to  the  acquisition 
organization.  Q3  Q4 

C6  All  delivered  subcontracted  and  vendor 
software  that  is  proposed  as  proprietary 
or  restricted  is  fully  justified  in 
accordance  with  governing  acquisition 
regulations;  e.g..  Federal  Acquisition 
Regulation  (FAR),  and  DOD  FAR 
Supplement.  Q5 


Ql  Identify  any  proprietary  or  restricted  rights 
software  including  reused  software  intended 
to  be  incorporated  into  the  delivered  system 
or  in  tools  to  be  used  to  develop  or  support 
the  delivered  system.  Provide  in  tabular 
form,  the  software  identification,  developer 
or  vendor,  and  description  of  the  associated 
restrictions.  Cl  C2  C4 

Q2  Identify  any  restricted  rights  that  would 

impact  the  ability  to  organically  support  the 
system  and  software  or  to  competitively 
procure  system  and  software  support  over 
the  life  cycle.  Describe  how  your  software 
development  process  accommodates  these 
restrictions.  Cl  C2  C3 

Q3  What  contractual  provisions  have  you 

instituted  with  your  subcontractors  to  obtain 
rights  sufficient  for  program  life  cycle 
support  and  maintenance?  Describe  how 
these  contractual  provisions  meet  your 
contractual  obligations  with  the  acquisition 
organization.  C5 

Q4  Where  licensing  agreements  are  required 
(e.g.,  on  software  tools),  are  they  being 
negotiated  for  the  total  system  program  life 
cycle?  C5 


Q5  Provide  for  each  proprietary  or  restricted 
rights  software  component  a  complete 
justification  as  to  why  this  software  is 
proprietary  or  restricted.  C6 


1 

1.5 

1.5.1 

Program  Management 

Risk  Control 

Risk  Identiflcation 

Cl 

Shortfalls  and  risks  associated  with  the 
proposed  development  activities  are 
identified.  Q1  Q2  Q3 

Qi 

Describe  your  process  to  identify  and 
reduce  technical  risks  associated  with  the 
system  and  software  development.  Cl 

C2 

Critical  paths  and  tasks  in  the  software 
development  and  associated  schedules  are 
identified  and  monitored.  Q4  Q5 

Q2 

Identify  the  projected  risks  and  shortfalls 
associated  with  this  program  as  a  result  of 
applying  this  process.  Cl 

Q3 

What  specific  risks  do  you  see  in  developing 
the  subject  program  software  with  the 
selected  design  methodologies  and 
implementation  languages?  Cl 

Q4 

Identify  the  critical  tasks  and  paths 
associated  with  the  proposed  development 
plan.  Describe  your  process  to  monitor 
these  critical  elements.  C2 

Q5 

Explain  how  you  apply  critical  path  and  risk 
management  techniques  in  managing 
software  schedules.  C2 
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1  Program  Management 

1.5  Risk  Control 

1.5.2  Risk  Management 


Cl  Risk  management  strategies  required  to 
identify  and  reduce  risk  are  identified 
consistent  with  the  program’s  cost, 
schedule,  and  performance  baselines.  Q1 

C2  Identified  risks  areas  are  tracked  and 
managed  throughout  the  program 
development.  Q2 

C3  Specific  criteria  are  identified  in  the  risk 
management  planning  applicable  to  each 
risk  reduction  activity.  These  criteria 
define  for  each  activity  the  condition 
under  which  each  risk  reduction  activity 
is  exercised.  Q3 

C4  Metrics  exist  for  management  tracking  of 
specific  program  risk  reduction  actions. 

Q4 


Q1  Describe  your  risk  management  process. 
What  role  will  prototypes  and 
demonstrations  play  in  risk  management? 

Cl 

Q2  Describe  how  identified  risk  areas  are 
analyzed,  tracked,  and  monitored 
throughout  the  program  development.  C2 

Q3  How  does  your  process  support  determining 
when  to  exercise  the  appropriate  risk 
reduction  activities?  Are  specific  criteria 
established  for  each  risk  reduction  activity? 
Are  variance  thresholds  established  for  each 
risk  area?  C3 

Q4  Identify  and  describe  the  metrics  to  be  used 
to  track  specific  program  risk  reduction 
activities.  C4 


A- 
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Figure  4.  Systems  Engineering 
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2 

2.1 

2.1.1 


Systems  Engineering 

System  Requirements  Development,  Management  and  Control 
Development  and  Allocation  of  Requirements 


Cl  A  systems  analysis  and  allocation  process 
is  used  to  verify  that  performance  and 
verification  requirements  are  correct  and 
complete  at  each  level  prior  to  further 
allocation  and  decomposition,  and  to 
verify  them  as  to  feasibility  and  top-level 
design  concept  prior  to  allocation  to 
software.  Ql 

C2  The  selected  systems  analysis  and 

allocation  methodology  is  compatible 
with  other  methodologies  adopted  on  the 
program.  Q2 

C3  System  requirements  (including  test  and 
verification  requirements)  are  analyzed, 
refined,  and  decomposed  to  ensure 
complete  functional  allocation  to 
hardware  and  software.  Q3 


C4  When  a  system-level  requirement  is 

allocated  to  more  than  one  configuration 
item  (Cl),  a  process  is  used  to  ensure  that 
the  lower-level  requirements  taken 
together  satisfy  the  system-level 
requirement.  Q4 

C5  A  defined  process  is  used  to  generate  the 
initial  versions  of  the  Software 
Requirements  Specifications  (SRS)  and 
the  Interface  Requirements  Specifications 
(IRS).  A  process  to  develop  and  review 
verification  requirements  for  each 
performance  requirement  is  in  place.  Q5 

C6  A  process  exists  to  identify  all  design 
documents,  requirements  specifications, 
and  interface  specifications  across  the 
development  team,  including 
subcontractors.  Q6 


Ql  How  are  system  and  subsystem  requirements 
defined  and  allocated?  How  are  these 
requirements  verified  at  each  level  prior  to 
further  allocation  and  decomposition?  How 
are  those  requirements  that  imply  digital 
processing  and  software  verified  as  to 
feasibility  and  top-level  design  concept 
prior  to  allocation  to  software?  Cl 

Q2  Describe  how  the  systems  analysis  and 

allocation  methodology  is  compatible  with 
the  systems  design  methodology,  and  with 
the  software  analysis  methodology.  C2 

Q3  Describe  the  process  by  which  system 
requirements  are  analyzed,  refined,  and 
decomposed  to  develop  a  functional 
allocation  to  hardware,  software,  and  other 
implementation  technologies.  Describe  the 
process  and  specific  trade  studies  and 
analyses  performed  to  aid  in  deciding  which 
requirements  to  allocate  to  hardware  and 
which  to  software.  C3 

Q4  Describe  the  process  ensures  that  when  a 
system-level  requirement  is  allocated  to 
more  than  one  Cl,  the  combination  of  the 
lower-level  requirements  meets  the  system- 
level  requirement.  C4 

Q5  Describe  the  process  used  to  generate  the 

SRS  and  IRS.  Describe  the  process  to  define 
verification  requirements  for  each 
performance  requirement  as  part  of  the 
requirements  and  definition  (specification 
preparation)  process.  C5 

Q6  Provide  a  tree  diagram  illustrating  the 

hierarchical  relationship  among  the  various 
levels  of  system,  subsystem,  critical  item,  and 
prime  item  requirements  specifications  and 
design  documents  down  to  and  including 
the  software.  This  tree  of  design  documents 
and  specifications  should  include  flowdown 
to  subcontracted  efforts.  C6 
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2 

2.1 

2.1.2 


Systems  Engineering 

System  Requirements  Development,  Management  and  Control 
Adequacy  of  Requirements 


Cl  System  requirements  are  analyzed  and 
refined  to  ensure  that  they  are  consistent, 
clear,  valid,  feasible,  compatible, 
complete,  and  testable,  and  they  do  not 
include  inappropriate  levels  of  design 
information.  Q1  Q2 

C2  Software  Requirements  Specifications 
(SRS)  and  Interface  Requirements 
Specifications  (IRS)  are  analyzed  and 
refined  to  assure  that  all  requirements 
allocated  to  software  are  adequately 
addressed,  and  that  they  do  not  include 
inappropriate  levels  of  design 
information.  They  are  reviewed  by  all 
affected  parties.  Q1  Q3 

C3  If  incremental  development  is  planned,  a 
process  is  used  to  establish  functional, 
performance,  and  verification 
requirements  for  each  incremental  system 
or  software  block/build.  This  process 
ensures  that  all  requirements  are  allocated 
to  planned  increments  prior  to  design 
and  development  of  the  increment.  Q4 


Q1  Describe  the  process  used  to  ensure  that 
system  requirements  are  consistent,  clear, 
valid,  feasible,  compatible,  and  complete. 
How  do  you  ensure  that  inappropriate  levels 
of  design  information  are  not  contained  in 
the  requirements  documents?  Cl  C2 

Q2  Describe  the  process  used  to  ensure  that 
system  requirements  have  complete 
verification  (test)  coverage.  Cl 


Q3  Describe  the  process  by  which  the  SRS  and 
IRS  are  analyzed  and  refined  to  ensure  that 
all  requirements  allocated  to  software  are 
adequately  addressed,  C2 


Q4  If  incremental  development  is  planned, 
describe  the  process  used  to  establish 
functional,  performance,  and  test 
requirements  for  each  incremental  system  or 
software  block/build.  Explain  how  these 
allocations  to  all  planned  blocks  are 
reviewed  and  baselined  prior  to  initiating 
design  and  development  of  the  first 
increment.  C3 
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2 

2.1 

2.1.3 


Systems  Engineering 

System  Requirements  Development,  Management  and  Control 
Requirements  Change  Control 


Cl  Requirements  are  baselined  early  in  the 
program  and  are  maintained  under 
configuration  control.  Q1 

C2  All  changes  to  requirements,  including 
those  generated  by  the  customer,  are 
managed  by  means  of  a  defined  change 
process.  Q2 

C3  Allocation  of  new  and  additional 
requirements  between  hardware  and 
software  is  managed  by  a  structured 
change  process;  reallocation  of  existing 
requirements  between  hardware  and 
software  is  managed  by  a  structured 
change  process.  Q3 


Qi  Describe  the  requirements  configuration 
management  process.  Cl 

Q2  Describe  the  requirements  change  control 
process,  with  reference  to  both  internally 
and  externally  generated  changes.  C2 

Q3  What  process  is  used  to  control  allocation  of 
changed  (new  or  existing)  requirements 
between  hardware  and  software?  C3 
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2 

2.1 

2.1.4 


Systems  Engineering 

System  Requirements  Development,  Management  and  Control 
Software  Impact  Analysis 


Cl  The  structured  change  process  for 

requirements  ensures  that  the  software 
impact  for  each  proposed  change  is 
addressed.  Q1 

C2  All  trade-off  studies  include  an 

assessment  of  the  software  impact  of  each 
alternative.  Trade  study  results  are 
documented  and  maintained  for  the  life 
of  the  program.  Q2  Q3 

C3  Software  is  addressed  in  all  systems 
engineering  reviews.  Q4 

C4  Areas  of  the  system  with  volatile 
requirements  are  monitored,  and 
requirements  specifications  are  reviewed 
for  ambiguities  that  could  result  in 
software  sizing  and  timing  instability  and 
other  program  impacts.  Q5 


Qi  How  is  the  software  impact  for  proposed 

changes  to  system  requirements  addressed? 
Cl 


Q2  What  process  is  used  to  include  the  software 
impacts  of  each  alternative  within  system- 
level  trade-off  studies?  C2 


Q3  How  are  trade-off  study  results 

documented?  How  are  they  maintained? 
C2 

Q4  What  provisions  exist  to  include  software 
issues  in  systems  engineering  reviews?  C3 


Q5  How  are  areas  of  the  system  with  volatile 
requirements  monitored?  Within  those 
areas,  how  is  the  impact  of  potential 
requirements  changes  to  the  program 
(including  software)  identified  and 
managed?  C4 
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.1 

.1.5 


Systems  Engineering 

System  Requirements  Development,  Management  and  Control 
Requirements  Traceability 


Cl  Two-way  requirements  traceability  is 

maintained  from  system  specifications  to 
hardware  and  software  configuration  item 
specifications.  Q1 


Qi  Describe  the  process  used  to  provide  two- 

way  requirements  traceability.  At  what  point 
is  requirements  traceability  established  and 
documented?  What  provisions  exist  to 
maintain  traceability?  Cl 
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2 

2.2 

2.2.1 


Systems  Engineering 

Computer  System  Architecture  Design  and  Review  Process 
Architecture  Deflnition 


Cl  A  process  exists  for  establishing  and 
maintaining  the  computer  system 
architecture  (hardware  and  software),  for 
determining  the  nature  and  number  of 
the  Computer  Software  Configuration 
Item  (CSCI),  and  for  maintaining 
traceability  of  the  architecture  to  system 
requirements.  Q1 

C2  A  process  is  used  to  define,  maintain,  and 
document  interfaces  (internal  and 
external)  within  the  architecture.  Q2 

C3  A  process  is  used  to  establish  and  show 
relationships  between  the  hardware  and 
software  components  within  the  computer 
system  architecture,  including  the  system- 
level  component  hierarchy  and  control 
structure.  Q3 

C4  A  process  is  used  to  establish  and  show 
relationships  between  the  computer 
system  architecture  (hardware  and 
software)  and  the  operational  (human) 
interface.  Q4 

C5  The  contractor  has  demonstrated  a 
commitment  to  applying  open  system 
architecture  concepts  in  their 
development.  Q5 


Ql  What  process  is  used  to  establish  and 

maintain  the  computer  system  architecture 
(hardware  and  software)?  Describe  how  the 
nature  and  number  of  CSCIs  is  defined. 
Describe  how  the  architecture  is  traced  to 
system  requirements.  Cl 


Q2  Describe  the  process  used  to  define, 

maintain,  and  document  interfaces  (internal 
and  external)  within  the  architecture.  C2 

Q3  What  process  is  used  to  establish  and  show 
relationships  between  the  hardware  and 
software  components  within  the  computer 
system  architecture,  including  the  system- 
level  component  hierarchy  and  control 
structure?  C3 

Q4  What  process  is  used  to  establish  and  show 
relationships  between  the  computer  system 
architecture  (hardware  and  software)  and  the 
operational  (human)  interface?  C4 

Q5  Describe  your  process  for  identifying, 
defining  and/or  selecting  interfaces  and 
protocols  that  will  be  used  between  the 
different  software  and  hardware  components 
of  the  system.  Will  any  of  these  interfaces 
be  proprietary?  C5 


2  Systems  Engineering 

2.2  Computer  System  Architecture  Design  and  Review  Process 

2.2.2  Adequacy  of  Architecture  Design 


Cl  A  process  exists  to  evaluate  suitability  of 
the  computer  system  architecture  for 
implementing  all  system  functional  and 
performance  requirements,  as  well  as  how 
design  constraints  are  satisfied.  Q1  Q2 
Q6 

C2  A  process  exists  to  evaluate  the  design 
based  on  use  of  risk  reduction 
techniques,  such  as  the  creation  of 
models  and  prototypes  (proofs, 
benchmarks).  Q4 


C3  A  process  exists  to  periodically  reassess 
the  adequacy  of  the  computer  system 
architecture  over  the  development  cycle. 

Q3  Q5 


Q1  Describe  who  participates  in  evaluating 
adequacy  of  the  computer  system 
architecture  design.  Cl 


Q2  What  process  is  used  to  evaluate  suitability 
of  the  computer  system  architecture  for 
implementing  all  system  functional  and 
performance  requirements?  Within  that 
process,  how  are  estimates  made  and 
budgeting  performed  regarding  the  use  of 
computer  system  resources?  Cl 

Q3  What  process  is  used  to  evaluate  suitability 
of  the  computer  system  architecture  for 
meeting  user  needs?  C3 

Q4  Describe  any  plans  for  using  risk  reduction 
techniques  such  as  creation  of  models  and 
prototypes  (proofs,  benchmarks).  C2 

Q5  What  is  the  process  for  reassessing  adequacy 
of  the  architecture  as  the  development  of  the 
system  progresses?  What  criteria  are  used  to 
either  stay  with  the  original  design  or 
change  it?  C3 

Q6  How  is  performance  of  the  architecture 
measured?  How  is  adequacy  of  the 
computational  resources,  memory,  processor 
capacity,  and  bus  bandwidth  established? 

Cl 


2 

2.2 

2.2.3 


Systems  Engineering 

Computer  System  Architecture  Design  and  Review  Process 
Architecture  Design  Review 


Cl  Whenever  system  requirements  change, 
there  is  a  review  of,  and  (as  necessary)  an 
update  to,  the  computer  system 
architecture  design.  Q1 

C2  The  computer  system  architecture  design 
is  reviewed  for  flexibility  to  adapt  to  new 
system  requirements.  Q2 


Ql  Describe  the  review  process  for  computer 
system  architecture  design.  Cl 


Q2  Explain  the  extent  to  which  design-for- 
change  considerations  and  flexibility  to 
adapt  to  new  system  requirements  are 
reviewed  on  the  program,  relative  to  the 
computer  system  architecture  design.  C2 
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2 

2.2 

2.2.4 


Systems  Engineering 

Computer  System  Architecture  Design  and  Review  Process 
Architecture  Change  Analysis 


Cl  There  is  a  review  of  all  architectural 
changes  and  their  impact  on  design 
margins  (such  as  memory,  throughput, 
bus  loading,  and  data  latency)  and  cost 
and  schedule  baselines.  Q1  Q2 


Q1  Describe  the  process  used  to  review  the 
impact  of  all  architectural  changes  on 
design  margins  (such  as  memory, 
throughput,  bus  loading,  and  data  latency). 
Cl 

Q2  Describe  the  process  used  to  review  the 

impact  of  all  architectural  changes  on  cost 
and  schedule  baselines.  Cl 
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2 

2.3 

2.3.1 

Systems  Engineering 
Supportability 

Reliability 

Cl 

Reliability  requirements  are  included  in 
system  requirements,  and  are  allocated  to 
hardware  and  software.  Q1 

Ql 

Describe  the  process  by  which  the  system 
reliability  requirements  are  allocated  to 
hardware  and  software.  Cl 

C2 

Reliability  is  defined,  measured, 
controlled,  and  reported  in  all  life-cycle 
phases.  A  process  is  used  to  institute 
corrective  actions  when  necessary.  Q2 

Q2 

How  is  reliability  in  all  life-cycle  phases 
defined,  measured,  controlled,  and  reported? 
C2 

2  Systems  Engineering 

2.3  Supportability 

2.3.2  Maintainability 


Cl  Maintainability  is  defined,  measured, 

Qi 

How  is  maintainability  defined,  measured. 

controlled,  and  reported.  A  process  is 
used  to  institute  corrective  actions  when 

controlled,  and  reported?  Cl 

necessary.  Q1  Q2  Q4 

C2  Support  systems  needed  for  any  required 

Q2 

How  is  maintainability  built  into  the  design? 

operational  self-sufficiency  are 
developed,  with  an  understanding 

Cl 

between  the  customer  and  the  developer 
regarding  the  effort,  cost,  and  equipment 

required  to  support  the  system.  Q3 

Q3 

Describe  the  process  used  to  evaluate  the 
effort,  cost,  and  equipment  needed  to 
support  the  system.  C2 

Q4 

Describe  the  process  to  manage  corrective 
actions.  Cl 
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2 

2.4 

2.4.1 


Systems  Engineering 
Intergroup  Coordination 
Group  Interfaces 


Cl  Throughout  the  development  life-cycle 
there  is  periodic  coordination  among 
developers,  acquisition  organizations, 
users,  maintainers,  and  testers  regarding 
user  needs,  acquisition  organization 
resources,  technology  status,  and  system 
requirements.  Requirements  changes 
resulting  from  interaction  with  users, 
maintainers,  and  testers  are  managed  with 
acquisition  organization  approval.  Q1 
Q2  Q3  Q4 

C2  There  is  a  systems  engineering  process 
which  (as  appropriate)  emphasizes  an 
integrated  product  development 
approach  and  defines  systems 
engineering  interfaces  with  other 
engineering  disciplines  and  development 
activities,  as  well  as  interfaces  between 
system  and  subsystem  developers.  Q5 
Q6  Q7 

C3  A  process  exists  to  manage,  provide  an 
escalation  path  for,  and  resolve  conflicts 
regarding  intergroup  issues,  including 
system-level  issues  that  arise  internally  or 
with  subcontractors.  Q8  Q9 

C4  Critical  dependencies  between 

development  groups  are  identified  and 
tracked.  QIO  Qll  Q12 


Ql  Describe  the  processes  to  be  followed  to 
have  users’  and  maintainers’  needs  and 
viewpoints  adequately  reflected  in  system 
requirements  throughout  development.  Cl 


Q2  Describe  the  processes  to  be  followed  to 
keep  system  requirements  in  balance  with 
acquisition  organization  resources 
throughout  development.  Cl 


Q3  Describe  the  processes  to  be  followed  to 
have  system  testers  adequately  involved  in 
the  requirements  definition  process 
throughout  development.  Cl 

Q4  Describe  the  processes  to  be  followed  to 
ensure  that  all  requirements  changes  take 
place  with  customer  approval.  Cl 

Q5  To  what  extent  is  an  integrated  product 

development  approach  to  be  followed?  C2 

Q6  How  will  systems  engineering  interface  with 
the  other  engineering  disciplines  and 
development  activities?  C2 

Q7  How  will  interfaces  between  the  various 
system  and  subsystem  developers  be 
managed?  C2 

Q8  Describe  the  processes  for  conflict 

resolution  to  be  used  internally  between 
development  groups.  C3 

Q9  Describe  the  processes  for  conflict 
resolution  to  be  used  between  prime 
contractors  and  subcontractors,  and  between 
subcontractors.  Describe  the  processes  used 
to  identify  and  resolve  intergroup  product 
interface  issues.  C3 
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BH 

Systems  Engineering 

Intergroup  Coordination 

Group  Interfaces  (Cont’d) 

QIO 

What  critical  dependencies  exist  between 
development  groups?  C4 

Qil 

Describe  the  processes  for  identifying  new 
critical  dependencies  during  development. 

C4 

Q12 

How  are  critical  dependencies  between 
development  groups  tracked?  C4 
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2 

2.4 

2.4.2 

Systems  Engineering 

Intergroup  Coordination 

Tool  Compatibility 

Cl 

Support  tools  used  by  the  different 
engineering  groups  enable  effective 
communication  and  coordination.  Q1 

Q1  Where  different  development  groups  have 
an  interface,  what  support  tools  will  be  used 
to  communicate  and  share  data?  Describe 
any  areas  of  potential  difficulty.  Cl 
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2  Systems  Engineering 

2.5  Systems  Engineering  Planning 

2.5.1  Methodology  and  Standards 


Cl  Detailed  systems  engineering  policies, 
practices,  and  procedures  are  defined, 
consistent  with  systems  engineering 
contractual  standards  such  as  MIL-STD- 
499B.  Q1 

C2  The  systems  engineering  process  makes 
provisions  for  documenting  the  rationale 
of  all  major  systems  engineering 
decisions.  Q2 

C3  A  process  is  used  to  arbitrate  contention 
across  trade-off  studies  for  utilization  of 
system-level  resources  and  reserves.  Q3 


Ql  Describe  how  the  program’s  systems 
engineering  policies,  practices,  and 
procedures  are  defined  and  documented. 

Cl 


Q2  Describe  the  provisions  that  have  been  made 
for  documenting  the  rationale  of  all  major 
systems  engineering  decisions.  C2 

Q3  What  policies,  practices,  and  procedures 
govern  system-level  trade-off  studies? 

When  there  is  contention  across  trade-off 
studies  for  utilization  of  system-level 
resources,  how  are  competing  claims 
resolved?  C3 
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2  Systems  Engineering 

2.5  Systems  Engineering  Planning 

2.5.3  Systems  Engineering  Master  Plan  (SEMP),  Master  Schedule  (SEMS),  and  Detailed 
Schedule  (SEDS) 


Cl  Systems  engineering  milestones 

(including  formal  reviews)  are  defined 
and  implemented  with  clear  completion 
criteria  in  the  SEMP/SEMS.  Q1 

C2  The  Software  Development  Plan  (SDP)  is 
coordinated  with  and,  as  appropriate, 
incorporated  into  the  SEMP/SEMS;  all 
software  milestones  are  accounted  for. 
Q2 


Q1  Describe  the  intended  use  of 

SEMP/SEMS/SEDS  on  the  program.  Are  all 
major  software  milestones  addressed  in  the 
SEMP,  SEDS  and  SEMS?  Are  completion 
criteria  specified  with  all  events?  Cl 

Q2  Describe  the  relationship  of  the  SDP  to  the 
SEMP/SEMS/SEDS.  C2 
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2 

2.5 

2.5.4 


Systems  Engineering 
Systems  Engineering  Planning 
Staffing 


Cl  Within  systems  engineering,  a  staffing  Q1  Is  there  a  staffing  plan  for  systems 
plan  defines  personnel  requirements,  engineering  on  the  program?  Have 

including  numbers,  skill  level,  experience,  required  software  skills  and  experience  been 

and  staffing  profile.  The  systems  identified?  Cl 

engineering  staffing  plan  identifies  the 
personnel  with  required  software  skills 
and  experience,  Q1 
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2  Systems  Engineering 

2.5  Systems  Engineeering  Planning 

2.5.5  Incremental  Development 

Cl  Incremental  software  development,  Q1  If  there  are  plans  for  incremental  software 

integration,  and  test  (with  a  series  of  development,  integration,  and  test,  describe 

builds),  if  used,  is  consistently  integrated  how  those  plans  are  coordinated  with  any 

into  plans  for  system  incremental  system  incremental  development  plans.  Cl 

development.  Q1 
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2  Systems  Engineering 

2.6  System  Integration  and  Test 

2.6.1  Integration  and  Test  Planning 

Cl  System  integration  planning  begins  in  the 
early  development  stages  of  the  program 
with  identification  of  the  person 
responsible,  and  includes  having  that 
person  participate  in  architecture  and 
design  reviews.  Q1 

C2  Test  planning  for  each  system  build 
includes  the  multiple  levels  of  system 
integration  and  test  (from  units  to 
Computer  Software  Configuration  Items 
(CSCIs)  to  subsystem  to  system-level 
test).  Q2 

C3  Any  incremental  software  development  is 
incorporated  into  system  integration  and 
test  planning.  Q3 

C4  Any  use  of  commercial-off-the-shelf 
(COTS)  software  or  other  reuse  software 
is  incorporated  into  system  integration 
and  test  planning.  Q4 

C5  System  integration  and  test  planning 
includes  verification  of  year  2000 
compliance  of  hardware  and  software. 
These  processes  are  adequately 
documented.  Q5 


Q1  When  will  the  person  responsible  for  system 
integration  planning  be  identified?  Does 
that  person  participate  in  architecture  and 
design  reviews?  Cl 


Q2  If  system  builds  are  planned,  describe  how 
test  planning  for  each  system  build  includes 
the  multiple  levels  of  system  integration  and 
test  (from  units  to  CSCIs  to  subsystem  to 
system-level  test).  C2 

Q3  If  incremental  software  development  is 

planned,  describe  how  it  is  incorporated  into 
system  integration  and  test  planning.  C3 

Q4  Describe  any  special  integration  and  test 

plans  developed  for  COTS  software  or  other 
reuse  software.  C4 


Q5  Describe  any  integration  and  test  plans 

developed  to  ensure  year  2000  compliance 
of  developed  software,  COTS  software,  other 
reuse  software,  and  hardware.  Where  are 
these  processes  documented?  C5 
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Systems  Engineering 

System  Integration  and  Test 

Test  Readiness 

Cl 

Readiness  criteria  are  clearly  identified 
for  formal  subsystem  and  system  test. 

Ql 

Ql 

Describe  the  process  to  be  used  to  ensure 
that  readiness  criteria  are  clearly  identified 
for  formal  subsystem  and  system  test.  Cl 

C2 

Readiness  and  completion  criteria  are 
clearly  identified  for  informal  subsystem 
and  system  test.  Q2 

Q2 

Describe  the  process  to  be  used  to  ensure 
that  readiness  and  completion  criteria  are 
clearly  identified  for  informal  subsystem 
and  system  test.  C2 

C3 

The  readiness  criteria  for  both  formal 
and  informal  test  are  incorporated  into 
the  Systems  Engineering  Master  Plan, 
Systems  Engineering  Master  Schedule, 
and  Systems  Engineering  Detailed 
Schedule  SEMP/SEMS/SEDS  milestones. 
Q3 

Q3 

How  are  these  readiness  criteria  reflected  in 
the  SEMP/SEDS/SEMS?  C3 
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2 

2.7 

2.7.1 


Systems  Engineering 
Reuse 

Opportunities  to  Reuse 


Cl  Opportunities  to  utilize  previously 
developed  system  and  software 
components  (including  architectures, 
designs,  code,  and  documentation)  are 
identified  and  subject  to  trade-off  studies 
Q1  Q2 


C2  Opportunities  to  utilize 

nondevelopmental  item  (NDI), 
commercial  off-the-shelf  (COTS),  and 
Government  furnished  equipment  (GFE) 
system  and  software  components 
(including  architectures,  designs,  code, 
and  documentation)  are  identified  and 
subject  to  trade-off  studies.  Q1  Q2 

C3  Opportunities  for  prime  contractors  and 
subcontractors  to  utilize  common  system 
and  software  components  in  different 
parts  (subsystems)  of  development  are 
identified  and  subject  to  trade-off  studies 
Q3 


Q1  What  previously  developed  system-level 
components  are  candidates  for  reuse  or 
adaptation  on  this  development?  What 
previously  developed  software  components 
are  candidates  for  reuse  or  adaptation  on 
this  development?  For  each,  indicate: 
source;  whether  NDI,  COTS,  or  GFE;  rights 
issues;  availability;  security  issues;  reliability 
level;  likelihood  of  use;  and  projected 
percent  of  modification  needed.  Cl  C2 

Q2  What  trade-off  studies  have  been  done  or 
are  planned  to  evaluate  the  costs,  benefits, 
and  risks  of  the  opportunities  to  reuse 
existing  system  and  software  components? 
Cl,  C2 


Q3  Describe  any  plans  to  identify  and  develop 
common  system-level  components  for 
shared  use  across  development  groups  or 
across  configuration  items  (CIs).  Describe 
any  plans  to  identify  and  develop  common 
software  components  for  shared  use  across 
development  groups  or  across  CIs.  Include 
both  intercompany  and  intracompany 
planning.  C3 
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2 

2.7 

2.7.2 


Systems  Engineering 
Reuse 

Life-Cycle  Issues 


Cl  Future  reuse  potential  of  newly 
developed  system  and  software 
components  is  maximized  within  the 
constraints  of  system  cost,  schedule,  and 
performance  baselines.  Q1  Q2 

C2  Life-cycle  cost  impact  of  reuse-related 
decisions  is  assessed,  including:  choice  of 
computer  languages,  processors, 
architectures,  and  environments; 
development  of  reusable  assets;  and 
maintenance  of  reuse  repositories.  Q3 

C3  The  contractor  has  a  process  in  place  for 
evaluating  the  applicability  of  reuse  code 
to  the  program’s  requirements  and 
ensuring  that  no  code  is  executed  for  a 
function  that  does  not  apply  to  the 
program  at  hand.  Q4 

C4  The  contractor  has  a  process  in  place  for 
monitoring  the  evolution  of  reuse  code 
and  ensuring  that  the  solution  is 
consistent  with  the  needs  of  the  program 
at  hand.  Q5 

C5  The  contractor  has  a  process  in  place  for 
ensuring  the  quality  of  reuse  code.  Q6 


Qi  Describe  plans  to  make  newly  developed 
system  and  software  components  reusable 
(generic,  adaptable).  Include  any  efforts  to 
perform  domain  engineering  tasks.  Describe 
the  impact  of  these  efforts  on  the  system 
and  software  baselines.  Cl 

Q2  Describe  the  impact  of  reuse  plans  on 
processes  used  to  perform  systems  and 
software  engineering  tasks  (such  as 
requirements  analysis,  design, 
implementation,  integration,  and  test).  Cl 

Q3  Describe  how  the  choice  of  computer 
platform,  software  architecture,  and 
development  environment  relates  to  plans  to 
create  and  maintain  reusable  system  or 
software  components.  C2 

Q4  Describe  your  process  for  evaluating 

applicability  of  all  components  of  reuse 
software  to  the  program  at  hand  and 
ensuring  that  no  extraneous  code  is 
executed  on  the  program.  C3 

Q5  Describe  your  process  for  ensuring  that 
upgrades  to  reuse  code  meet  the  needs  of 
the  program  in  terms  of  requirements  and 
schedule.  C4 

Q6  Describe  your  process  for  ensuring  that  the 
quality  of  the  reused  code  meets  the  needs 
of  the  program.  C5 
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2  Systems  Engineering 

2.7  Reuse 

2.7.3  Resource  Management 

Cl  Processes,  procedures,  and  tools  exist  to 
document,  manage,  and  control  reusable 
components.  Q1  Q2  Q3  Q4 

Qi 

Explain  whether  reusable  components  are 
documented  differently  from  the  rest  of  the 
system.  Cl 

Q2 

Explain  whether  reusable  components  are 
managed  and  controlled  differently  from 
the  rest  of  the  system.  Cl 

Q3 

Explain  whether  any  special  tools  are 
needed  to  manage  reusable  components. 

Cl 

Q4 

Explain  whether  any  special  processes  or 
procedures  are  needed  to  manage  reusable 
components.  Cl 
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Functional  Area  3:  Software  Engineering 


Figure  5.  Software  Engineering 


3 

3.1 

3.1.1 


Software  Engineering 
Software  Development  Planning 
Software  Estimating 


Cl  Estimates  for  size,  effort,  cost,  and 

schedule  of  each  software  component  are 
generated  according  to  a  documented 
procedure.  Estimates  for  incrementally 
developed  software  are  generated 
consistently  with  published  methods  and 
company  experience.  Q1 


C2  Estimates  for  manpower  profiles  required 
for  completion  of  estimated  schedules  of 
each  software  product  are  generated 
according  to  a  documented  procedure. 
Estimated  profiles  are  consistent  with  past 
actual  company  experience.  Q2 


C3  Estimates  for  required  critical  computer 
resources  needed  by  each  software 
component  are  generated  according  to  a 
documented  procedure.  Q3 


C4  As  software  system  definition  progresses, 
estimates  are  performed  for  each  lowest- 
level  software  component,  and  previous 
estimates  are  revised.  Q4 


C5  Environmental  parameters  and 

calibration  factors  applied  in  estimating 
manpower  requirements,  and  the 
resulting  productivity  factors,  are 
consistent  with  past  actual  productivity 
rates  for  similar  applications.  Q5 


Qi  How  are  estimates  for  the  size,  effort,  cost, 
and  schedule  of  each  software  component 
generated?  Which  published  estimating 
methods  and  models  are  used?  Describe 
how  estimates  are  developed  for  any 
planned  incremental  development  or 
release.  Describe  your  experience  with  this 
method  relative  to  actual  size,  effort,  cost, 
and  schedule  of  completed  projects.  Cl 

Q2  How  are  effort  profiles  estimated  for 

software  components?  What  process  ensures 
that  each  task  required  for  the  software 
development  (i.e.,  requirements,  definition, 
analysis,  design,  code,  integration,  and  test) 
are  included  in  the  estimates?  How  are  the 
cost  and  effort  of  related  software 
engineering  tasks  (such  as  configuration 
management,  quality  assurance,  and  test  and 
integration)  included  in  estimates  for  the 
program?  C2  C6 

Q3  How  are  estimates  generated  for  required 
critical  computer  resources  needed  by  each 
software  component?  How  are  computer 
resources  estimated  and  balanced  across  the 
program  to  ensure  that  critical  needs  are 
met?  C3 

Q4  Describe  how  estimates  are  revised  as 

software  system  definition  progresses.  What 
process  ensures  that  estimates  remain 
consistent  with  the  current  state  of  the 
overall  program?  What  is  the  approach  to 
revising  estimates  as  lowest-level  software 
components  are  comprehensively  defined? 
By  what  program  milestone  are  these  revised 
estimates  baselined?  C4 

Q5  How  are  the  various  environmental 

parameters  and  calibration  factors  derived? 
What  is  the  source  of  productivity  factors 
applied  in  estimating  manpower 
requirements?  Are  they  consistent  with  past 
actual  productivity  rates?  Are 
environmental  parameters  and  calibration 
factors  for  use  in  estimating  under 
configuration  management?  C5 
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3 

3.1 

3.1.1 


Software  Engineering 
Software  Development  Planning 
Software  Estimating  (Cont’d) 


C6  The  estimating  methods  account  for  all 
tasks  and  steps  associated  with  the 
software  development  process  to  be  used. 

Q2 


C7  All  data  required  to  repeat  the  above 
estimates  for  each  software  component 
are  recorded  and  maintained.  Q6  Q7 


C8  The  estimating  process  ensures 

consistency  among  estimates  for  size, 
workload  effort,  distribution  of 
manpower,  schedule,  and  cost.  Q8 

C9  Software  estimates  are  periodically 

compared  to  actual  results  to  calibrate  the 
estimating  models  and  procedures.  Q9 


Q6  How  is  the  data  required  to  repeat  the  above 
estimates  for  each  software  component 
recorded  and  maintained?  Is  the  data 
configuration  controlled  and  available  to  all 
who  need  it?  Are  occasional  audits  done  to 
verify  that  the  required  data  is  accurate  and 
available?  C7 

Q7  Who  is  responsible  for  development  and 
storage  of  the  above  estimates?  Who 
ensures  that  estimates  are  performed 
according  to  procedure,  and  that  the  data  is 
recorded  and  maintained?  C7 

Q8  Describe  the  methodology  for  correlating 
and  ensuring  consistency  among  estimates 
of  size,  workload  effort,  distribution  of 
manpower,  schedule,  and  cost.  C8 

Q9  How  is  accuracy  of  software  estimates 
ensured?  Is  there  a  formal  process  for 
periodically  calibrating  software  estimating 
procedures  with  actual  performance  data? 
C9 


3  Software  Engineering 

3.1  Software  Development  Planning 

3.1.2  Software  Work  Packages 


Cl  Software  components  and  work  packages 
of  manageable  size  and  development 
effort  are  defined  to  enable  management 
of  the  entire  software  system.  Q1 


C2  The  defined  set  of  software  work 

packages  is  used  to  manage  the  work 
tasks  associated  with  software 
development.  Q2 


C3  Manpower  is  allocated  to  individual  work 
packages  consistent  with  their  individual 
development  schedules.  In  addition, 
software  development  manpower  is 
allocated  consistent  with  the  total  software 
development  needs  of  the  program.  Q3 


Ql  How  is  the  overall  software  effort  organized 
into  manageable  software  components? 
What  factors  are  considered  in  determining 
the  appropriate  size  and  development  effort 
for  each  component?  How  is  the  software 
organization  documented?  Cl 

Q2  How  are  software  work  packages  planned 
and  defined?  Describe  the  criteria  for 
acceptable  software  work  packages.  Explain 
how  the  software  work  package  is  used  to 
manage  the  work  (i.e.,  plan,  define,  assign 
resources  and  responsibility,  and  status  and 
report  progress).  C2 

Q3  Describe  your  method  for  allocating 
manpower  to  individual  software  work 
packages  as  well  as  across  the  total  software 
development  effort.  How  does  this 
allocation  method  apply  to  incremental 
(block,  build)  software  development?  How 
does  the  method  provide  realistic  manpower 
profiles,  based  on  experience,  to  support 
total  program  needs?  C3 
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3 

3.1 

3.1.3 


Software  Engineering 
Software  Development  Planning 
Software  Development  Process  Model 


Cl  A  well-defined  Software  Development 
Process  Model  has  been  selected  for  use 
on  the  program  at  hand.  Q1 


C2  The  criteria  used  to  select  and/or  define 
the  Software  Development  Process  Model 
are  sound  and  consistent  with  the  needs 
of  the  program  at  hand.  Q2 


Qi  Describe  the  Software  Development  Process 
Model  selected  for  the  program:  what 
activities  it  comprises,  how  activities  are 
sequenced  and  iterated,  what  are  the 
entrance  and  exit  criteria  from  one  activity 
to  the  next,  and  from  one  iteration  to  the 
next.  Cl 

Q2  Provide  the  rationale  for  selecting  the 

Software  Development  Process  Model  and 
its  associated  risks.  C2 
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3  Software  Engineering 

3.1  Software  Development  Planning 

3.1.4  Preparing  the  Software  Development  Plan  (SDP) 


Cl  The  SDPs  are  developed  and  maintained 
using  a  sound  and  complete  process.  Q1 


C2  A  process  exists  for  coordinating  SDPs 
across  team  members  and  ensuring 
integrity  and  supportability  of  the 
program’s  software.  Q2  Q3 


Qi  Describe  the  process  used  to  generate  the 

SDPs  for  the  project,  who  participates  in  the 
effort,  where  results  are  recorded,  and  how 
plans  are  maintained.  Describe  the  software 
development  planning  that  will  be 
performed;  what  software  development  life- 
cycle  activities  will  be  covered  in  SDPs;  how 
the  plan  accounts  for  processes,  schedules, 
and  manpower  needed  to  develop  the 
software  to  be  delivered.  Cl 

Q2  Explain  how  you  will  ensure  that  the 
software  subcontractors’  SDPs  are 
compatible  with  the  prime  contractor’s 
SDPs.  How  will  software  integrity  and 
supportability  across  software  developed  by 
the  prime  contractor  and  subcontractors  be 
ensured?  C2 

Q3  Which  components  of  the  software 

development  organizations  (across  the 
prime  contractor  and  subcontractors) 
coordinate  on  the  final  contents  of  the  SDPs 
and  how  do  they  do  that?  How  are  terms, 
dependencies,  and  responsibilities 
negotiated  and  communicated  internal  to 
the  prime  contractor,  between  the  prime 
contractor  and  subcontractors,  and  among 
subcontractors?  C2 
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.2 

.2.1 


Software  Engineering 

Software  Project  Tracking  and  Reporting 

Software  Tracking 


Cl  The  size,  effort,  cost,  and  schedule  status  Q1  How  often  will  engineering  and  program 

of  each  software  work  package  is  management  measure  and  review  the  size, 

periodically  measured  and  reviewed  by  effort,  cost,  and  schedule  status  of  each 

engineering  management  and  corrective  software  component?  What  criteria  and 

actions  are  taken  when  pre-established  conditions  will  trigger  corrective  actions? 

variance  thresholds  are  exceeded.  Ql  How  will  success  of  corrective  actions  be 

measured?  What  provisions  exist  for  event- 
driven  engineering  management  reviews? 
Cl 

C2  The  critical  computer  resources  required  Q2  How  often  will  critical  computer  resources 

by  and  allocated  to  each  software  work  required  by  each  of  the  software 

package  are  periodically  measured  and  components  be  measured  and  reviewed  by 

reviewed  by  management,  and  corrective  engineering  and  program  management? 

actions  taken  when  pre-established  When  will  it  be  deemed  necessary  to  take 

variance  thresholds  are  exceeded.  Q2  corrective  action?  Who  is  responsible  for 

setting  variance  thresholds?  C2 
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3  Software  Engineering 

3.2  Software  Project  Tracking  and  Reporting 

3.2.2  Software  Reporting 


Cl  The  status  of  each  software  work  package 
is  reported  to  all  involved  levels  of 
engineering  and  program  management 
through  periodic  reporting  up  the  chain 
of  command.  Qi 


C2  Development  process/performance  and 
product  quality  measurements  are 
recorded,  analyzed,  and  used  for 
improving  process  and  product  quality 
on  the  program.  These  data  are  recorded 
and  maintained  for  organizational 
process  and  product  quality 
improvements.  Q2 


Ql  How  is  the  status  of  each  software  work 
package  reported  up  the  chain  of 
command?  What  specific  elements  of 
software  status  (  e.g.,  units,  components, 
configuration  items,  subsystem,  system)  are 
reported  to  each  management  level  from 
first-level  supervisor  through  program 
manager?  What  situation,  condition,  or 
threshold  would  trigger  a  status  report  to  a 
higher  level  of  management  than  normally 
would  be  necessary  for  a  work  package? 

Cl 

Q2  What  actual  measurements  of  development 
performance  and  product  quality  will  be 
recorded  during  software  development? 
How  will  these  measurements  be  analyzed 
and  used  for  changing  and  improving 
products  and  processes?  How  will  metrics 
be  recorded  and  maintained?  Who  is 
responsible  for  collection,  storage,  and 
analysis  of  metrics?  C2 
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3  Software  Engineering 

3.3  Software  Requirements  Management 

3.3.1  Software  Requirements  Analysis 


Cl  Software  requirements  are  analyzed  for 
completeness,  correctness,  clarity, 
feasibility,  and  verifiability.  Q2  Q3 


C2  Requirements  derived  from  the  Software 
Requirements  Specification  are 
documented  and  maintained.  Q4 


C3  The  selected  requirements  analysis 

methodology  is  compatible  with  other 
methodologies  applied  on  the  program. 
The  analysis  methodology  is  supported 
with  necessary  tools.  Q1  Q5 


Qi  Describe  the  software  analysis  process  to  be 
applied.  Identify  specific  methodologies 
and  tools  to  be  used  to  support  the  analysis 
process.  What  organizational  element  is 
responsible  to  perform  the  analysis? 

Identify  the  input  to  and  output  product 
from  the  analysis.  C3 

Q2  How  are  the  software  requirements  analyzed 
for  completeness,  correctness,  etc.?  How  do 
you  determine  that  the  software 
requirements  are  complete,  adequate,  and 
verifiable?  Cl 

Q3  How  are  the  total  sets  of  software 

requirements  analyzed  as  a  whole,  including 
interfaces?  Cl 


Q4  If  additional  requirements  are  derived  from 
the  baselined  requirements,  where  are  they 
documented?  How  are  they  maintained? 
How  is  their  impact  on  cost  and  schedule 
determined?  C2 

Q5  Describe  the  methodology  used  to  analyze 
requirements.  Is  it  compatible  with  the 
requirements  traceability  methodology?  Is 
it  compatible  with  the  design  methodology? 
Is  it  compatible  with  the  development 
language?  C3 
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.3 

.3.2 


Software  Engineering 
Software  Requirements  Management 
Software  Requirement  Changes 


Cl  Software  development  artifacts 
(requirements,  design,  code  and 
documentation)  are  revised  as  changes  to 
the  requirements  are  incorporated.  Q1 

C2  As  changes  and  additions  to  requirements 
are  incorporated.  Software  Development 
Plans  (SDPs)  and  program  baselines  (cost 
and  schedule)  are  reviewed  and  modified 
if  necessary.  Q2 

C3  Two-way  traceability  between  software 
and  system  requirements  is  established 
and  maintained.  Q3 


Ql  Describe  the  software  development  activities 
resulting  from  a  change  in  or  addition  to 
requirements.  When  are  they  performed? 
How  do  you  ensure  that  they  are 
performed?  Cl 

Q2  Describe  the  software  planning  activities 
resulting  from  a  change  in  requirements. 
When  are  they  performed?  How  do  you 
ensure  that  they  are  performed?  C2 

Q3  How  do  you  ensure  that  two-way  traceability 
between  system  requirements  and  software 
requirements  is  maintained?  C3 
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3  Software  Engineering 

3.4  Software  Design 

3.4.1  Design  Methodology 


Cl  A  methodology  is  used  to  develop, 

document,  and  maintain  top-level  and 
detailed  software  design.  Q1 


C2  The  design  description  includes  the 
(static)  structure  and  the  (dynamic) 
behavior  of  the  software.  Q2  Q3  Q4 

C3  The  design  description  includes  all 
software  interfaces.  Q5 


C4  The  estimated  use  of  the  target  computer 
resources  is  refined  and  included  in  the 
design  documentation.  Q6  Q7 

C5  Plans  and  mechanisms  exist  to  document 
the  design  decisions  including  trade-off 
studies.  Q8 


C6  The  selected  design  methodology  is 
compatible  with  other  methodologies 
adopted  on  the  program.  Q9  QIO 


C7  The  taxonomy  used  to  represent  design, 
code,  and  test  entities  are  either  common 
across  entities  or  are  compatible  and 
mapped.  Q7  QIO 


C8  The  contractor  has  demonstrated  a 
commitment  to  basing  their 
implementation  on  a  sound  software 
architecture.  Qll 


Q1  Describe  the  process  and  specific 

methodologies  used  to  develop  the  top-level 
and  detailed  software  design.  Is  the  same 
methodology  used  to  maintain  the  design 
through  development  and  life  cycle 
support?  What  tools  are  used  to  support  the 
methodology?  Cl 

Q2  What  mechanism  and  format  are  used  to 

describe  the  components  of  the  design?  C2 


Q3  What  mechanism  and  format  are  used  to 
describe  the  execution  priorities  of  the 
different  components,  and  the  execution 
control?  C2 

Q4  How  is  the  data  flow  described?  C2 


Q5  How  does  the  design  methodology  describe 
the  interfaces  internal  to  the  software 
components?  How  does  it  describe  the 
interfaces  between  the  software 
components?  How  does  it  describe  the 
interfaces  between  the  software  and  other 
components  of  the  system?  C3 

Q6  Describe  the  timelines  generated  to 
represent  CPU  usage  of  the  target 
computer(s).  How  are  these  timelines 
maintained  with  the  design  documentation? 

C4 

Q7  Describe  the  memory  maps  (or  other 
representations  of  memory  utilization) 
generated  to  represent  the  memory  usage  of 
the  target  computer(s).  How  are  they 
maintained  with  the  design  documentation? 
C4  Cl 

Q8  How  are  the  design  decisions  documented 
and  communicated?  How  are  the  trade-off 
results  analyzed?  How  is  this  process 
enforced?  C5 

Q9  Is  the  design  methodology  compatible  with 
the  requirements  analysis  methodology?  Is 
it  compatible  with  the  development 
language?  C6 
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3 

3.4 

3.4.1 


Software  Engineering 

Software  Design 

Design  Methodology  (Cont’d) 


QIO  Is  the  same  taxonomy  (Computer  Software 
Components,  units,  packages,  files,  etc.)  used 
to  represent  the  design  entities,  the  code 
entities,  the  test  entities  and  the 
configuration  management  entities?  If  not, 
how  is  the  mapping  between  them  defined? 
C6  Cl 

Qll  Describe  the  phase  of  your  development 
that  deals  with  software  architecture,  the 
resources  allocated,  the  processes  and  tools 
that  will  be  used  and  the  architecture  outputs 
that  will  be  generated.  C8 
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.4 

.4.2 


Software  Engineering 
Software  Design 
Design  Assurance 


Cl  Mechanisms  exist  to  ensure  the 

modularity,  cohesiveness,  feasibility,  and 
coupling  of  the  design.  Q1  Q2  Q3 


C2  Exit  criteria  are  used  to  ensure 

completeness  and  quality  of  the  design 
before  it  is  baselined.  Q4 


C3  Two-way  traceability  between  design  and 
requirements  is  established  and 
maintained.  Q5 


Qi  Describe  the  exit  criteria  for  baselining  the 
top-level  design.  Do  they  include  adherence 
to  the  design  methodology?  What  are  the 
criteria  for  completeness  of  the  design 
documentation?  How  are  the  development 
standards  required  for  the  program  reflected 
in  the  exit  criteria?  How  are  they  enforced? 
Cl 

Q2  Describe  the  exit  criteria  for  baselining  the 
detailed  design.  Do  they  include  adherence 
to  the  design  methodology?  What  are  the 
criteria  for  completeness  of  the  detail  design 
documentation?  How  are  the  development 
standards  required  on  the  program  reflected 
in  the  exit  criteria?  How  are  they  enforced? 
Cl 

Q3  How  is  the  correctness,  completeness,  and 
feasibility  of  the  design  ensured?  Who  has 
that  responsibility?  What  tools  are  used?  Is 
it  part  of  the  exit  criteria  described  above? 
Identify  the  design  quality  attributes 
included  in  the  exit  criteria.  Cl 

Q4  How  are  the  modularity,  cohesiveness,  and 
coupling  of  the  design  ensured?  Who  has 
that  responsibility?  What  tools  are  used?  Is 
it  part  of  the  exit  criteria  described  above? 
Identify  the  design  characteristics  included 
in  the  exit  criteria.  C2 

Q5  How  is  traceability  established  from  the 
requirements  to  the  design,  and  from  the 
design  to  the  requirements?  At  what  point 
in  the  design  is  it  done,  and  by  whom?  How 
is  it  documented?  How  is  it  maintained? 
What  tools  are  used?  Is  this  traceability  part 
of  the  exit  criteria  described  above?  C3 
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3 

3.5 

3.5.1 


Software  Engineering 
Software  Coding  and  Unit  Testing 
Code  Development 


Cl  The  program  software  coding  standards 
contain  guidelines  regarding  internal 
documentation,  style,  complexity,  and  use 
of  language  features.  Q1  Q2  Q3  Q4 

C2  Use  of  the  target  computer’s  resources  is 
measured  and  compared  to  budgeted 
values.  Corrective  action  is  taken  as 
required.  Q5  Q6 

C3  The  developed  software  is  unit  tested. 
Realistic  resources  and  schedules  are 
allocated  to  this  level  of  testing.  Units  are 
tested  in  all  increments  of  development. 

Q7  Q8 

C4  The  software  is  reviewed  against  the 
design,  and  2-way  traceability  between 
software  code  and  design  is  established 
and  maintained.  Q9 

C5  Exit  criteria  exist  for  establishing  that 
each  lowest-level  software  unit  has  been 
implemented  correctly,  is  performance 
tested,  and  conforms  with  the  coding 
standards.  QIO 


Qi  Describe  the  process  for  ensuring  that 

coders  use  a  common  coding  style  and  that 
the  code  is  sufficiently  and  uniformly 
documented.  Cl 

Q2  Explain  how  complexity  of  code  is 

minimized.  What  guidelines  are  followed? 
What  tools  are  used?  Cl 


Q3  Identify  any  guidelines  for  use  of  special 

features  of  the  development  language.  How 
are  they  communicated?  Cl 


Q4  Describe  any  plans  to  use  code  generation 
technology  for  this  development.  What  are 
the  potential  benefits,  risks,  and  life-cycle 
trade-offs?  Cl 

Q5  How  are  timing  measurements  made?  At 
what  component  level?  How  often?  Who 
has  that  responsibility?  How  are  the  results 
used  ?  C2 


Q6  How  are  memory  usage  measurements 
taken?  At  what  component  level?  How 
often?  Who  has  that  responsibility?  How 
are  the  results  used  ?  C2 

Q7  What  processes  and  procedures  are  used  to 
ensure  that  the  design  is  implemented 
completely  and  correctly?  At  what 
component  level?  Who  has  that 
responsibility?  C3 

Q8  How  do  you  ensure  that  each  lowest-level 
software  component  (unit)  is  unit  tested? 
How  do  you  ensure  that  adequate  resources 
are  allocated  and  adequate  schedule 
(duration)  is  planned  to  support  this  level  of 
testing?  What  tools  are  used?  Who  has  that 
responsibility?  C3 


Q9  How  is  traceability  from  the  software  code 
to  the  design  and  from  the  design  to  the 
code  established  and  maintained?  When  is  it 
done?  How  is  it  documented  and 
maintained?  What  tools  are  used?  Who  has 
that  responsibility?  C4 


.5 

.5.1 


Software  Engineering 
Software  Coding  and  Unit  Testing 
Code  Development  (Cont’d) 


QIO  What  exit  criteria  exist  for  establishing  that 
each  lowest-level  software  unit  is  ready  for 
integration?  Do  they  include  compliance 
with  coding  standards?  Do  they  include 
peer  reviews?  Do  they  include  unit  testing? 
Do  they  include  conformance  to  the  design? 
How  are  they  enforced?  C5 
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.5 

.5.2 


Software  Engineering 
Software  Coding  and  Unit  Testing 
Code  Changes 


Cl  Code  changes  are  unit  tested  before  they 
are  incorporated.  Q1 

C2  Code  changes  are  reviewed  for 

correctness,  and  to  avoid  undesired 
impact  on  other  software  and  system 
variables  and  components.  Q2  Q3 


Q1  Describe  your  process  for  ensuring  that  all 
code  changes  are  unit  tested.  Cl 

Q2  Describe  your  process  for  estimating  the 

effect  of  code  changes  on  other  parts  of  the 
system,  including  variables  and  other 
software  components.  What  tools  are  used? 
Who  is  involved  in  the  process?  C2 

Q3  Describe  your  process  for  reviewing  code 
changes  after  the  code  has  been  baselined. 
Are  they  reviewed  for  correctness  and 
compliance  with  the  standards?  What  if 
there  are  urgent  changes?  How  is  the  review 
process  enforced?  Who  has  that 
responsibility?  C2 
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3 

3.6 

3.6.1 


Software  Engineering 
Software  Integration  and  Test 
Software  Integration 


Cl  The  software  integration  planning  takes 
into  account  the  interdependencies 
between  the  different  software 
components  and  the  criticality  of  each 
component.  Q1  Q2  Q3 


C2  The  software  integration  planning  takes 
into  account  the  availability  of  other 
components  of  the  system.  Q1  Q4 

C3  For  planned  incremental  software 
development,  software  integration  is 
planned,  scheduled,  and  resources  are 
allocated  to  support  each  increment  of 
software  development.  Q5 

C4  The  software  integration  planning  and 
process  accommodate  software 
integration  starting  with  the  lowest-level 
elements  (i.e.,  units  through  all  levels, 
including  Computer  Software 
Configuration  Items  (CSCIs)  and  CSCI/ 
Hardware  Configuration  Items  (HWCIs). 

Qi 


Qi  Describe  your  process  for  planning  software 
integration.  How  many  components  do  you 
integrate  at  once?  How  do  you  determine 
the  order  for  integrating  the  components? 
Describe  how  your  integration  process 
accommodates  all  levels  of  software 
integration.  Cl  C2  C4 

Q2  How  are  the  dependencies  between  software 
components  determined?  At  what  level? 
How  does  it  affect  integration  planning?  Cl 

Q3  How  is  the  criticality  of  each  component 
determined?  What  role  does  it  play  in 
integration  planning?  Cl 


Q4  How  does  your  integration  planning  handle 
situations  where  a  needed  software  or 
hardware  component  is  not  available  on 
time?  C2 


Q5  How  does  your  integration  planning  cover 
integration  with  all  planned  software 
increments  (blocks,  builds)?  C3 
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.6 

.6.2 


Software  Engineering 
Software  Integration  and  Test 
Software  Testing 


Cl  The  software  test  process  includes 

development  of  test  plans,  procedures, 
and  test  cases.  Ql 

C2  A  process  exists  to  ensure  that  software 
testing  is  adequately  planned  with 
sufficient  test  resources.  Q2 


C3  An  approach  is  used  that  plans  for  all 
levels  of  testing  to  ensure  thorough 
testing  of  the  software.  Q3  Q4 


C4  A  process  exists  for  incorporating 

changes  resulting  from  software  testing. 
Q5  Q6 


C5  A  regression  test  methodology  ensures 
that  system  performance  is  maintained 
after  revisions  are  made  to  the  software 
components.  Q7  Q8 

C6  Software  testing  is  planned  with  adequate 
schedules  and  resources  for  all  planned 
software  development  increments  (blocks, 
builds).  Q9 


Ql  How  are  test  plans,  procedures  and  cases 

developed?  When  ?  By  whom?  Where  are 
they  documented?  How  are  they  reviewed? 
How  are  they  controlled?  Cl 

Q2  What  tools  will  be  used  for  testing?  When 

will  they  be  available?  Will  they  require  any 
special  inputs?  Will  their  outputs  require 
any  special  processing?  What  is  your 
process  to  ensure  that  all  required  test 
resources  have  been  planned  and  allocated? 
C2 

Q3  Does  your  software  test  and  verification 
process  define  specific  levels  of  software 
test?  What  are  they?  How  do  they  relate  to 
the  structure  of  your  software  design?  C3 

Q4  What  are  the  completion  criteria  for  each 

level  of  testing?  Do  you  generate  test  plans 
and  procedures  for  each  level?  If  so,  how 
are  they  coordinated  across  levels?  C3 

Q5  Describe  your  process  for  incorporating 
changes  resulting  from  errors  uncovered 
during  testing.  Where  is  it  documented? 
How  is  it  enforced?  C4 

Q6  Does  your  change  process  ensure  that 
changes  are  incorporated  into  the  right 
baseline  version?  Are  there  provisions  for 
priority  changes  requiring  quick 
turnaround?  C4 

Q7  What  is  your  process  for  regression  testing? 
Are  there  guidelines  for  when  and  how 
regression  tests  should  be  run?  Is  regression 
testing  factored  into  the  schedules?  C5 

Q8  Is  there  a  library  of  regression  tests?  If  so, 
describe  how  it  is  generated.  Are  any 
regression  testing  tools  used?  C5 

Q9  How  is  the  software  test  process  and 
discipline  applied  to  each  planned 
increment  (block,  build)  of  software 
developed?  C6 


Functional  Area  4:  Quality  Management  and  Product  Control 


4.8  Documentation 
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Figure  6.  Quality  Management  and  Product  Control 
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4 

4.1 

4.1.1 

Quality  Management  and  Product  Control 

Software  Quality  Management 

Quality  Planning 

Cl 

The  program’s  software  quality  plan  is 
the  basis  for  its  activities  for  software 
quality  management.  Q1  Q2  Q4 

Qi 

Describe  your  process  for  developing  a 
quality  plan  for  the  program.  Cl 

C2 

The  program’s  software  quality  plan 
contains  provisions  to  ensure  that  quality 
is  built  into  the  software.  Q3 

Q2 

How  are  the  activities  described  in  the 
program  quality  plan  reflected  in  proposed 
work  packages?  Cl 

C3 

The  program’s  quality  goals  and  their 
priorities  for  the  software  products  are 
defined,  monitored,  and  revised 
throughout  the  software  life  cycle.  Q5 

Q3 

What  provisions  does  the  quality  plan  have 
to  ensure  that  quality  is  built  into  the 
software?  C2 

C4 

The  plan  identifies  points  in  the  process 
where  software  quality  is  measured.  Q6 

Q4 

Is  quality  training  planned  for  the 
development  organizations?  Is  it 
mandatory?  What  does  it  consist  of?  Cl 

C7 

C5 

The  plan  identifies  methods  for 
analyzing  the  program’s  quality 
measurements,  for  evaluating  whether 
they  meet  the  customer’s  needs,  and  for 
determining  necessary  corrective  actions. 
Q7 

Q5 

Does  the  quality  plan  define  specific  quality 
goals  for  each  software  product?  Does  it 
describe  how  these  goals  are  prioritized? 

Does  it  describe  how  these  goals  are 
monitored  and  kept  consistent  with  the 
customer’s  needs?  C3 

C6 

The  software  program’s  quality  goals  for 
the  products  are  allocated  appropriately 
to  the  subcontractors  delivering  software 
products  to  the  program.  Q8 

Q6 

Are  there  checkpoints  for  measuring 
software  quality  identified  in  the  quality 
plan?  C4 

Cl 

Plans  exist  for  members  of  development 
organizations  to  receive  required  training 
in  software  quality.  Q4 

Q7 

Does  the  quality  plan  describe  how  the 
quality  data  is  analyzed  and  how  it  is  used? 

C5 

Q8 

Have  the  program’s  quality  goals  been 
incorporated  into  subcontracts  where 
appropriate?  C6 
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4. 

4.1 

4.1.2 

Quality  Management  and  Product  Control 

Software  Quality  Management 

Product  Evaluations 

Cl 

Independent  product  evaluations  are 
performed  for  all  software  work  products 
before  they  are  baselined.  Q1  Q2  Q3 

Q4  Q5 

Ql 

Do  you  have  evaluation  procedures  for  all 
the  software  products  to  be  developed  on 
the  program?  Do  they  describe  the 
evaluation  criteria  in  sufficient  detail?  Cl 

C2 

Responsibility  for  each  product 
evaluation  is  clearly  defined.  Q5 

Q2 

When  are  the  software  products  evaluated? 

Cl 

Q3 

Are  adequate  resources  (cost  and  schedule) 
provided  for  them?  Cl 

Q4 

What  are  the  evaluator’s  qualifications?  Do 
they  receive  any  special  training?  Cl 

Q5 

Is  responsibility  for  each  product  evaluation 
defined?  At  what  level?  Cl  C2 
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4 

4.1 

4.1.3 

Quality  Management  and  Product  Control 

Software  Quality  Management 

Software  Discrepancies 

Cl 

Specific  procedures  exist  to  resolve 
software  versus  hardware  discrepancies 
and  to  identify,  document,  track,  and 
resolve  software  discrepancies.  Q1  Q2 

Q3  Q4 

Ql 

How  are  software  discrepancies  managed? 

Cl 

Q2 

Identify  and  describe  specific  procedures  to 
identify,  document,  report,  track,  and  resolve 
software  discrepancies.  Cl 

Q3 

Describe  your  method  for  resolving 
software  versus  hardware  discrepancies  in 
your  problem  reporting  systems.  Cl 

Q4 

Describe  tools  used,  and  any  automation,  to 
address  discrepancy  collections,  tracking, 
and  reporting.  Cl 
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4 

4,2 

4.2.1 

Quality  Management  and  Product  Control 

Software  Quality  Assurance  (SQA) 

SQA  Organizational  Approach 

Cl 

An  organization  is  assigned  the 
responsibility  to  monitor  the  software 
development  process  and  the  software 
products.  Q1 

Ql 

Describe  the  responsibilities  of  the  SQA 
organization  and  how  it  interfaces  with  other 
organizations.  Cl  C2 

C2 

The  responsibilities,  mission,  and 
interface(s)  of  SQA  with  the  engineering, 
configuration  management,  and  test 
functions  are  defined  and  documented. 

Q1  Q2  Q3 

Q2 

Does  the  the  SQA  organization 
communicate  the  results  of  SQA  activities  to 
the  engineering  organization?  C2 

C3 

The  SQA  group  is  empowered  to  effect 
changes  to  the  program  when  quality 
goals  are  not  followed.  Q4  Q5 

Q3 

How  does  the  SQA  function  interface  with 
engineering,  configuration  management, 
and  test  functions?  C2 

Q4 

What  can  the  SQA  organization  do  if  the 
software  development  process  and 
procedures  are  not  being  followed?  C3 

Q5 

What  mechanisms  and  channels  exist  for 

SQA  to  surface  quality  problems  and  elevate 
them  in  the  management  chain  until  they 
are  resolved?  C3 
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H 

Quality  Management  and  Product  Control 

Software  Quality  Assurance  (SQA) 

Compliance  Checking 

Cl 

The  program  follows  a  written  SQA  plan 
for  measuring  and  monitoring  the 
performance  of  the  program’s  defined 
software  process.  Q1 

Q1 

Where  are  SQA  activities  defined  for  the 
program?  C 1 

C2 

Adherence  to  defined  software 
development  and  management  processes 
is  verified.  Q2 

Q2 

Describe  how  SQA  ensures  compliance  of 
software  development  activities  with  defined 
processes.  Which  processes  are  audited? 

How  often?  C2 

C3 

SQA  audits  designated  software  work 
products  to  verify  compliance  with 
quality  goals  and  adherence  to  the 
applicable  standards  and  requirements. 

Q3Q4 

Q3 

Describe  how  SQA  ensures  compliance  of 
software  management  activities  with  planned 
processes.  Which  processes  are  audited? 

How  often?  C3 

Q4 

Describe  how  SQA  verifies  that  the  software 
products  adhere  to  the  program’s 
requirements,  standards,  and  quality  goals. 

C3 
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4 

4.3 

4.3.1 

Quality  Management  and  Product  Control 

Defect  Control 

Defect  Activity  Coordination 

Cl 

The  program  develops  and  maintains  a 
plan  for  its  defect  prevention  activities. 

Qi 

Qi 

Describe  your  program  plan  for  preventing 
software  defects.  Cl 

C2 

Revisions  to  the  standard  software  process 
resulting  from  defect  prevention  actions 
are  incorporated.  Q2 

Q2 

Are  defect  causes  assessed  for  potential 
process  improvement  and  incorporation  into 
program  and  organizational  development 
processes?  C2 

i 

Quality  Management  and  Product  Control 

Defect  Control 

Defect  Collection  and  Analysis 

Cl 

Common  causes  of  defects  are  identified, 
prioritized,  and  systematically  eliminated. 

Q1  Q2  Q3 

Qi 

Describe  your  approach  to  collection  and 
analysis  of  defects.  Cl  C2 

C2 

Causal  analysis  meetings  are  conducted. 

Q1Q4 

Q2 

Does  your  program  have  a  process  for 
identifying  common  causes  of  defects?  Cl 

C3 

Data  on  defects  identified  in  peer  reviews, 
document  review,  and  testing  are 
collected  and  analyzed.  Q5 

Q3 

Are  known  common  causes  of  defects 
prioritized  for  correction?  Cl 

Q4 

Are  causal  analysis  meetings  conducted? 
Define  your  procedures  for  conducting 
causal  analysis  meetings.  C2 

Q5 

Identify  your  approach  to  collecting  defects 
resulting  from  peer  reviews,  testing,  and 
design  reviews.  Is  this  approach  contained  in 
the  quality  plan?  C3 
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4  Quality  Management  and  Product  Control 

4.3  Defect  Control 

4.3.3  Defect  Reporting 

Cl  Defect  prevention  data  are  documented  Q1  Is  defect  prevention  data  documented  and 
and  tracked  across  the  teams  that  are  communicated  across  all  teams  participating 

coordinating  defect  prevention  activities.  in  defect  prevention  activities?  Does  this 

Ql  information  include  feedback  on  status  and 

results  of  defect  prevention  activity?  What  is 
the  frequency  of  this  communication?  Cl 
C2 

C2  Members  of  the  software  engineering 
group  and  software-related  groups 
receive  feedback  on  status  and  results  of 
the  organization’s  and  program’s  defect 
prevention  activities  on  a  periodic  basis. 

Ql 
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4  Quality  Management  And  Product  Control 

4.4  Metrics 

4.4.1  Metrics  Collection  And  Definition  Process 


Cl  The  metrics  selected  and  the  strategy  for 
the  data  collection  and  the  analyses  to  be 
performed  are  determined  based  on  the 
program’s  defined  software  process.  Q1 
Q2  Q3 

C2  The  specific  measusrement  data  to  be 
collected,  their  precise  definitions,  the 
intended  use  and  analysis  of  each 
measurement,  and  the  process  control 
points  at  which  they  will  be  collected, 
reported,  and  fed  back  are  defined.  Q2 
Q3 

C3  The  established  metrics  process  includes 
the  requirement  to  define  variance 
thresholds,  which  when  broken,  require 
corrective  action.  Q4 


Q1  Describe  your  process  for  defining  the 
metrics  that  will  be  used  on  this  program 
and  determining  their  use.  Cl 


Q2  Describe  (for  each  collected  metric)  how  it 
will  be  collected,  the  points  at  which  it  will 
be  collected,  how  it  will  be  analyzed,  how  it 
will  be  reported,  and  to  which  organizations 
it  will  be  reported.  Cl  C2 


Q3  Describe  the  processes  and  tools  used  for 
data  collection  and  analyses.  Cl  C2 


Q4  Describe  your  use  of  variance  thresholds. 
Describe  how  these  thresholds  are 
established  and  used  in  development 
management.  C3 
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Quality  Management  and  Product  Control 

Metrics 

Metrics  Selected  for  the  Program 

Cl 

The  metrics  selected  for  the  program 
address  the  system  and  software  products, 
the  process  used  to  generate  the  products, 
and  the  progress  of  the  development 
effort.  Q1 

Ql 

Identify  the  metrics  you  plan  to  collect  on 
this  program,  which  system  or  software 
product  they  apply  to,  which  process  they 
apply  to,  and/or  what  progress  they  measure. 
Cl 

C2 

The  measurement  program  is  integrated 
with  the  program’s  development  process 
across  the  life-cycle  and  teaming 
arrangements.  Q2  Q3  Q4 

Q2 

Describe  (for  each  collected  metric)  what 
life-cycle  phase  it  applies  to.  C2 

Q3 

Describe  the  various  organizations  that  will 
be  involved  in  the  measurement  program, 
what  their  role  will  be,  and  how  they  will 
interface.  C2 

Q4 

How  are  these  measurements  defined  and 
integrated  with  your  program’s  defined 
software  process?  C2 
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4  Quality  Management  and  Product  Control 

4.5  Peer  Reviews 

4.5.1  Peer  Review  Planning 


Cl  Internal  documents  exist  that:  identify 
required  participants  in  the  reviews, 
provide  specific  criteria  for  successful 
completion,  and  describe  documentation 
required  for  the  review  and  how  follow- 
on  actions  are  documented,  tracked,  and 
controlled.  Ql 

C2  Peer  reviews  are  planned  consistent  with 
the  peer  review  internal  standards  and 
procedures.  Q2 

C3  Peer  review  plans  specify  the  schedule  of 
peer  reviews.  Q2 

C4  The  peer  review  schedule  is  consistent 
with  the  Systems  Engineering  Master 
Plan  and  Systems  Engineering  Master 
Schedule  (SEMP/SEMS).  Q2 


Ql  Describe  the  documented  internal  peer 
review  procedures  and  requirements 
including  definition  of  required  participants, 
completion  criteria  and  review  content,  and 
follow-on  action  item  resolution.  Cl 


Q2  Describe  how  peer  reviews  are  planned  and 
scheduled.  Describe  how  the  peer  review 
schedule  is  consistent  with  other  program 
schedules  (e.g.,  SEMP/SEMS).  C2  C3  C4 
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I 

Quality  Management  and  Product  Control 

Peer  Reviews 

Peer  Review  Performance 

Cl 

Peer  reviews  are  performed  according  to 
the  peer  review  plan.  Q1 

Qi 

Describe  how  peer  reviews  are  performed 
according  to  the  peer  review  plan.  Cl 

C2 

Reviews  are  documented  (i.e.,  review 
process,  requirements,  conduct,  and 
results).  Q2 

Q2 

Describe  how  peer  review  results  are 
documented  and  to  whom  results  are 
distributed.  C2  C3 

C3 

Review  results  are  reported  to  appropriate 
managers.  Q2 
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4 

4.6 

4.6.1 


Quality  Management  and  Product  Control 
Internal  Independent  Verification  and  Validation  (IIV&V) 
IIV&V  Planning 


Cl  A  well-defined  systematic  approach  to 

IIV&V  is  documented  in  accordance  with 
contract  requirement.  Q1 

C2  The  planning  process  for  IIV&V 

includes  criteria  to  select  elements  of  the 
software  for  which  the  IIV&V  process  is 
applicable.  Q2 

C3  Elements  of  the  software  to  which  the 
IIV&V  process  will  be  applied  are 
selected.  Q2 

C4  Required  resources  and  tools  are 
identified  in  IIV&V  planning.  Q2 


Qi  Describe  your  process  for  planning  and 
conducting  IIV&V.  Cl 


Q2  Do  you  have  a  procedure  for  determining 
which  software  should  undergo  IIV&V? 

Has  this  procedure  been  applied  to 
identification  of  software  for  IIV&V  for  this 
program?  C2  C3  C4 
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Quality  Management  and  Product  Control 
.6  Internal  Independent  Veriflcation  and  Validation  (IIV&V) 

.6.2  Technical  Evaluation  and  Implementation  Process 


Cl  The  IIV&V  function  is  independent  of 
the  software  development  function.  Q1 

Qi 

What  is  the  organizational  responsibility  and 
reporting  chain  for  the  program  IIV&V? 

Cl 

C2  Sufficient  resources  and  required  tools 
are  available  to  accomplish  the  IIV&V 
process.  Q2 

Q2 

Describe  the  resources  and  tools  available  to 
accomplish  the  IIV&V  process.  C2 
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4 

4.7 

4.7.1 


Quality  Management  and  Product  Control 
Software  Configuration  Management  (SCM) 
Planning 


Cl  A  sound  process  is  in  place  for  planning 
the  program’s  SCM,  as  well  as  for 
disseminating  and  implementing  the 
plans  (draft).  Q1 


C2  The  SCM  is  integrated  with  other 

configuration  management  plans  used  on 
the  program  (draft).  Q2 


Qi  Describe  SCM  planning  for  this  project. 
What  guidance  exists  for  development, 
maintenance,  and  distribution  of  this  SCM 
planning  and  the  procedures  to  implement 
the  plan?  Who  reviews  the  plan  and 
procedures?  Cl 

Q2  Explain  how  SCM  is  integrated  with  system 
configuration  management,  engineering 
management,  and  other  development 
disciplines.  C2 


4  Quality  Management  and  Product  Control 

4.7  Software  Configuration  Management  (SCM) 

4.7.2  Baseline/Configuration  Identiflcation  and  Management 


Cl  The  SCM  process  includes  identifying 
and  controlling  master  copies  of  the 
products  of  the  software  development 
process,  including  baselines  and  test 
configurations,  for  each  block  or  build. 
The  mechanism  and  media  for  storing 
the  master  copies,  such  as  libraries  and 
other  tools,  is  appropriate  for  maintaining 
required  configuration  control.  The 
process  also  controls  the  release,  and  re- 
release  after  revision,  of  the  master 
copies.  Ql 

C2  The  software  work  products  to  be  placed 
under  SCM  are  identified,  including 
baselines  and  test  configurations  for  each 
block  and/or  build  called  for  by  the 
developer’s  software  development 
process  model.  Q2 

C3  Procedures  are  specified  for  the  unique 
identification  (by  name  and  number)  of 
each  product  placed  under  configuration 
control.  This  uniqueness  is  applied  across 
all  software  development  processes 
invoked  across  all  functional  disciplines 
and  prime  and  subcontractors  involved  in 
the  project.  Q3 


Ql  Explain  how  products  of  the  software 

development  process,  including  baselines 
and  test  configurations,  are  controlled, 
released,  revised,  and  rereleased  for  each 
block  or  build.  Cl 


Q2  Which  products  of  the  software  development 
process  will  be  placed  under  configuration 
control?  C2 


Q3  How  will  consistent  software  identification 
across  the  program’s  functional  disciplines 
and  the  prime  and  subcontractors  be 
maintained?  How  will  revisions  be 
identified  for  each  release?  C3 
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4  Quality  Management  and  Product  Control 

4.7  Software  Configuration  Management  (SCM) 

4.7.4  Conflguration  Control  and  Status  Accounting 

Cl  Change  control  procedures,  which  Q1  Describe  the  change  control  procedures  to 

include  the  equivalent  of  configuration  be  followed  for  changes  requested  to 

control  boards  for  software,  are  defined  products  of  the  software  development 

and  integrated  into  the  program  change  process  under  configuration  control.  Cl 

management  process.  Q1 

C2  The  software  development  process  is  used  Q2  If  you  anticipate  the  use  of  object  code 

to  manage  and  control  the  use  of  object  patches,  describe  how  it  will  be  managed 

code  patches  at  all  levels  and  ensure  that  and  controlled,  including  when  a  patch  will 

software  patches  are  a  rare  exception  and  be  incorporated  into  a  baseline  and  when  it 

are  always  resolved  with  permanent  will  be  deleted.  C2 

changes.  The  use  of  patches  is  approved 
and  documented  by  the  configuration 
control  board.  Q2 

C3  Change  requests  and  problem  reports  for  Q3  Explain  how  configuration  status 

all  products  of  the  software  development  accounting  will  be  accomplished  on  this 

process  under  configuration  control  are  project.  C3 

recorded,  reviewed,  approved,  and 

tracked.  Libraries  and  other  tools  are 

incorporated  into  the  configuration 

management  and  control  function 

support  status  accounting.  C3 
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4  Quality  Management  and  Product  Control 

4.7  Software  Configuration  Management  (SCM) 

4.7.5  Configuration  Management  Library  and  Tools 


Cl  SCM  tools  are  adequate,  proven,  and  Q1 

available  for  use  on  the  project  and  for 
transition  to  the  post-deployment  support 
environment.  Included  in  the  tools  is  the 
equivalent  of  a  controlled  access  library 
system  which  is  in  use  and  has 
procedures  for  software  and 
documentation  releases  and  for 
promotions  to  other  libraries.  Q1 

C2  For  efficiency  and  reliability  of  SCM  and  Q2 
control,  the  SCM  and  control  process  and 
tools  are  integrated  with  the  tools  used  to 
create  the  products  of  the  software 
development  process.  Q2 


Describe  the  SCM  and  control  tools  to  be 
used  on  this  project,  your  experience  with 
them,  and  your  plans  to  transition  them  to 
the  post-deployment  environment.  Cl 


Describe  how  SCM  and  control  process  and 
tools  are  integrated  with  software 
development  process  activities  and  the 
software  engineering  environment.  C2 
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4 

4.8 

4.8.1 


Quality  Management  and  Product  Control 
Documentation 

Documentation  Identification,  Production,  and  Control 


Cl  A  periodic  update  based  on  reviews  is 
held  with  the  acquisition  organization, 
user,  and  support  organizations  regarding 
the  design  information  to  be  delivered. 

Qi 

C2  All  internal  and  deliverable 

documentation  products  are  clearly 
identified,  including  draft 
documentation.  Q2 

C3  The  documentation  used  to  operate  and 
maintain  the  software  is  developed  and 
maintained  consistent  with  the  current 
software  baseline.  Q3 

C4  Documentation  is  integrated  into  the 

engineering  development  process  and  the 
system,  subsystem,  and  hardware 
documentation.  Q4 


C5  Requirements  for  subcontracted  software 
documentation  are  tied  to  the  Systems 
Engineering  Master  Schedule  (SEMS) 
and  Systems  Engineering  Master  Plan 
(SEMP).  Q5 

C6  Software  documentation  is  integrated  into 
the  software  development  process  and  is 
supported  by  the  System/Software 
Engineering  Environment  (S/SEE).  Q6 

C7  Automated  tools  support  documentation 
generation.  Q7 


C8  The  documentation  is  accessible  from  the 
S/SEE.  Q8 


Qi  Describe  your  process  to  define  the  detailed 
information  to  be  included  in  the  design 
documentation.  How  are  the  needs  of 
acquisition  organizations,  users,  and 
supporters  accommodated  in  this  process? 
Cl 

Q2  Describe  the  process  by  which  the  necessary 
deliverable  documentation  is  identified. 

How  does  incremental  development  affect 
documentation  development?  C2 

Q3  How  is  documentation  developed  and 
maintained?  What  process(es)  ensure 
accuracy  and  completeness?  C3 

Q4  How  is  software  documentation  integrated 

into  your  software  engineering  development 
process?  Describe  the  role  of 
documentation  in  support  of  engineering 
activities.  C4 

Q5  Describe  your  requirements  for 

subcontracted  software  documentation.  Are 
these  requirements  tied  to  the  SEMS  and 
SEMP?  C5 


Q6  Is  the  key  software  documentation 
accessible  from  the  S/SEE?  C6 


Q7  Identify  any  automated  tools  that  support 
the  generation  of  software  documentation. 
Is  the  software  technical  requirements  and 
development  information  accessible  from 
the  S/SEE?  C7 

Q8  Does  the  key  software  documentation  exist 
in  the  S/SEE?  What  procedure  ensures  that 
the  S/SEE  contains  the  current 
documentation  baseline?  C8 
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4 

4.8 

4.8.2 


Quality  Management  and  Product  Control 

Documentation 

Technical  Adequacy 


Cl  Systems  engineering  decisions, 
development  rationale,  and  test 
information  are  captured  and  retained  in 
the  documentation.  Qi 

C2  Standards  exist  for  documenting  test 
requirements  for  the  software.  Q2 

C3  Internal  standards  or  requirements  for 
software  documentation  that  are 
integrated  with  system,  subsystem  and 
hardware  documentation  are  consistent 
with  the  requirements  of  the  contract. 

Q3 

C4  Documentation  completion  is  a  integral 
part  of  internal  reviews  to  ascertain 
software  development  status  and 
progress.  Documentation  requirements 
are  part  of  the  completion  criteria  for  the 
Systems  Engineering  Master  Plan  and 
Systems  Engineering  Master  Schedule. 
Q4 

C5  Consistency  and  currency  is  maintained 
across  software  work  products  including 
software  plans,  process  descriptions, 
allocated  requirements,  software 
requirements,  software  design,  code,  test 
plans,  and  test  procedures.  Q5 


Ql  Does  the  program  have  a  process  for 

capturing  engineering  decisions?  Describe 
your  approach  for  documenting  design 
decisions,  development  rationale,  and  test 
information.  Cl 

Q2  What  standards  do  you  use  in  documenting 
test  requirements  documents?  C2 

Q3  Is  the  documentation  approach  integrated 
with  engineering  activities?  How  does  your 
software  documentation  integrate  with 
system,  subsystem,  and  hardware 
documentation?  C3 


Q4  What  role  does  documentation  play  in  your 
internal  reviews  to  ascertain  software 
development  status  and  progress?  Do 
completion  criteria  for  formal  and  informal 
reviews  include  appropriate  documentation? 
C4 


Q5  Is  consistency  maintained  across  software 
products  from  requirements  through 
acceptance  testing  (i.e.,  traceability  across 
software  requirements,  software  plans, 
design,  code,  and  test)?  What  ensures  that 
this  is  accomplished?  C5 
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Functional  Area  5:  Organizational  Resources  and  Program  Support 


Figure  7.  Organizational  Resources  and  Program  Support 
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5 

5.1 

5.1.1 


Organizational  Resources  and  Program  Support 

Organizational  Standards  and  Procedures 

System  and  Software  Development  Processes  (Cont’d) 


Q5  Describe  your  approach  for  version  control 
and  controlling  changes  to  the 
organization’s  standard  system  development 
and  software  development  process(es).  How 
do  you  know  which  version  of  the 
organization’s  standard  is  in  use  at  a  given 
time?  How  are  changes  to  the  standard 
assessed,  incorporated  within  the  standard, 
and  incorporated  by  the  program?  C4 
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5  Organizational  Resources  and  Program  Support 

5.1  Organizational  Standards  and  Procedures 

5.1.2  Tailoring 


Cl  A  waiver  procedure  and  tailoring  Q1  Describe  any  documented  guidelines 

guidelines  and  criteria  are  available  to  provided  for  tailoring  organizational 

facilitate  tailoring  the  organization’s  standards  to  specific  program  requirements, 

standard  systems  development  software  What  specific  program  needs  require 

development  process(es)  to  meet  specific  tailoring  on  this  program?  How  was  the 

program  requirements  and  needs.  Q1  specific  system  development  and  software 

Q2  development  model  for  this  program 

selected?  Given  the  systems  and  software 
development  model  for  this  program,  how 
are  the  organization’s  system  and  software 
development  processes  and  procedures 
tailored  to  be  compatible  with  and  support 
the  development  model?  Cl 

C2  When  organizational  standards  are  Q2  Describe  the  procedure  for  waiving 

applied  directly  to  programs,  a  process  compliance  with  the  organization’s  standard 

exists  to  verify  compliance  with  these  system  development  and  software 

standards.  Q3  development  process(es).  How  does  it 

support  application  of  the  tailoring 

[If  there  is  more  than  one  standard  guidelines?  Describe  how  the  procedure 

process,  answer  the  questions  for  each  provides  flexibility  for  those  cases  where 

standard  applicable  to  the  program.]  particular  program  needs  require  extensive 

tailoring.  Cl 

Q3  How  is  compliance  by  program  with  the 

applicable  organization’s  standard  process 
verified?  How  is  compliance  by  program 
with  the  activity  descriptions  within  the 
standard  ensured  or  verified,  taking  into 
account  approved  deviations?  How  is 
proper  function  of  the  interfaces  within  and 
between  the  various  systems  and  software 
disciplines  ensured  or  verified,  taking  into 
account  approved  deviations?  C2 
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5  Organizational  Resources  and  Program  Support 

5.1  Organizational  Standards  and  Procedures 

5.1.3  Capturing  and  Making  Available  Use  Information 


Cl  Past  use  data  for  standard  organizational 
and  program  processes  is  collected.  This 
data  includes  estimates  and  actuals, 
quality  measurements,  peer  review/test 
coverage  and  efficiency,  number  and 
severity  of  defects  found.  This 
experience-based  data  is  made  available 
to  programs  for  planning  and  managing 
new  programs.  Ql 

C2  A  library  of  process-related 

documentation  (e.g.,  program  standards, 
measurement  plans,  process  training 
materials)  is  maintained  and  made 
available  to  the  program  to  support  reuse 
of  proven  processes  and  interpretation  of 
usage  data.  Q2 

[If  there  is  more  than  one  standard 
process,  answer  the  questions  for  each 
standard  applicable  to  the  program.] 


Ql  Explain  how  data  from  use  of  the 

organization’s  and  programs’  development 
processes  and  resulting  products  is  collected 
and  made  accessible  to  the  program  for  use 
in  planning  and  managing  its  effort.  In 
addition  to  actual  measurement  data,  what 
related  information  is  maintained  to  help  the 
program  understand  and  interpret  the 
measurement  data  and  assess  it  for 
reasonableness  and  applicability?  Cl 

Q2  For  the  program,  what  process-related 
documentation  is  maintained  and  made 
available  to  support  reuse  of  proven 
processes  and  interpretation  of  usage  data? 
How  are  these  documentation  items 
catalogued  for  easy  access?  C2 
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5  Organizational  Resources  and  Program  Support 

5.2  Facilities 

5.2.1  Development  Facilities 


Cl  A  plan  for  establishing  and  maintaining 
the  required  system  and  software 
development  facilities  exists,  and  is 
consistent  with  the  program’s 
requirements,  needs,  usage  estimates,  and 
schedule.  Q1  Q2  Q3  Q4 


C2  Where  required,  integrated 
systems/software  development 
environments  are  planned  and  acquired 
or  developed  to  be  in  place  to  meet 
program  need  dates.  Q5 

C3  Planning  for  system  and  software 

development  facilities  includes  support  of 
all  planned  incremental  development 
(blocks,  builds),  including  regression 
testing.  Q6 

C4  For  team  developments  (including 

primes,  associates,  and  subcontractors) 
common  and  compatible  development 
facilities  are  planned  to  ensure  continuity, 
integrity,  and  supportability  of  developed 
systems  and  software.  Q7 


Ql  Describe  the  software  development  facilities 
(host  development  computers,  workstations, 
networks,  memory  systems,  etc.)  intended 
for  the  program  in  terms  of  quantity, 
location,  availability  date,  capacity  and 
response  time.  Describe  the  level  of 
integration  of  the  system/software 
development  facilities  (environments).  Cl 

Q2  Describe  the  basis  for  determining  that  the 
facilities  will  satisfy  the  program’s 
requirements  and  needs  (capabilities  and 
capacities).  Cl 


Q3  Describe  when  each  facility  will  be  available. 
What  is  your  plan  to  ensure  that  the  facilities 
will  be  available  to  meet  the  program’s  need 
dates?  What  is  your  fallback  position  if  any 
of  these  facilities  are  not  available  in  time 
for  the  program?  Cl 

Q4  Describe  how  the  development  facilities  will 
be  maintained  for  the  program.  Cl 


Q5  For  integrated  software  environments, 
describe  which  systems  and  software 
functions  are  integrated  and  how  these 
integrated  facilities  will  be  planned  and 
acquired  or  developed  to  meet  program 
needs.  C2 

Q6  Describe  how  the  software  development 
facilities  are  planned  and  acquired  to 
support  incremental  software  development 
(blocks,  builds).  C3 

Q7  For  team  developments  (e.g.,  primes, 

associates,  and  subcontractors)  describe  how 
compatible  development  facilities  are 
planned  to  ensure  continuity,  integrity,  and 
supportability  of  the  developed  systems  and 
software.  C4 
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5  Organizational  Resources  and  Program  Support 

5.2  Facilities 

5.2.2  Specialized  Facilities 


Cl  The  specialized  facilities  needed  for  the 
program,  such  as  special  test  stations  and 
simulation  laboratories,  have  been 
identified  and  meet  the  program’s  needs. 
A  systematic  process  is  used  to  ensure 
that  all  required  facilities  are  identified, 
planned,  and  acquired  or  developed  to  be 
in  place  to  meet  program  need  dates.  Q1 

C2  Availability  of  specialized  facilities  is 
consistent  with  the  program’s 
requirements,  needs,  usage  estimates,  and 
schedule.  Q2  Q3  Q5 


C3  A  plan  to  support  these  specialized 
facilities  during  the  life-cycle  of  the 
program  exists  and  is  consistent  with  the 
program’s  resources  and  schedule.  Q4 

Q6 

C4  Where  required,  integrated 

systems/software  specialized  facilities  are 
planned  and  acquired/developed  to  be  in 
place  to  meet  program  need  dates.  Q7 


C5  Planning  for  system/software  specialized 
facilities  includes  support  of  all  planned 
incremental  development  (blocks,  builds), 
including  regression  testing.  Q6 

C6  For  team  developments  (including 

primes,  associates)  and  subcontractors, 
compatible  specialized  facilities  are 
planned  to  ensure  continuity,  integrity, 
and  supportability  of  developed  systems 
and  software.  Q7 


Q1  Describe  the  specialized  testing  facilities, 
simulation  laboratories,  and  any  other 
specialized  facilities  that  will  be  used  on  the 
program.  How  do  you  ensure  that  all 
required  facilities  are  identified,  planned, 
and  acquired  or  developed  to  be  in  place  to 
meet  program  need  dates?  Cl 


Q2  Describe  your  plans  for  acquiring  or 

developing  specialized  facilities,  if  they  are 
not  already  in  place.  What  is  your  fallback 
position  if  any  of  these  facilities  are  not 
available  in  time  for  the  program?  C2 

Q3  Describe  the  process  for  scheduling  the  use 
of  these  facilities  and  the  number  of  shifts 
per  day  scheduled  for  each.  How  do  you 
ensure  that  their  availability  will  meet 
program  needs  in  terms  of  number  of  users 
and  development  schedule?  C2 

Q4  Describe  how  each  specialized  facility  will 
be  supported  during  the  life-cycle  of  the 
program,  in  terms  of  user  support  and 
maintenance.  Where  is  this  support  included 
in  the  program’s  allocated  personnel  and 
cost  baselines?  C3 

Q5  For  integrated  specialized  facilities,  describe 
which  systems  and  software  functions  are 
integrated  and  how  these  integrated  facilities 
will  be  planned  and  acquired  or  developed 
to  meet  program  needs.  C2 

Q6  Describe  how  the  system  and  specialized 

software  facilities  are  planned  and  acquired 
to  support  incremental  software 
development  (blocks,  builds).  C5 


Q7  For  team  developments  (e.g.,  primes, 

associates,  and  subcontractors)  describe  how 
compatible  specialized  facilities  are  planned 
to  ensure  continuity,  integrity,  and 
supportability  of  developed  systems  and 
software.  C4  C6 
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5 

5.3 

5.3.1 


Organizational  Resources  and  Program  Support 

Training 

Training  Plans 


Cl  A  program  training  plan  exists  that 
identifies: 

•  the  program’s  current  and  future 
technical,  management,  and  skill 
needs 

•  how  these  needed  skills  will  be 
developed  (informal  vehicles, 
formal  courses  that  need  to  be 
developed  or  procured  from 
outside  sources) 

•  the  resources  (e.g.,  trainers, 
materials,  funding,  time)  needed  to 
develop  these  skills 

•  the  schedule  for  required  training 

Qi 

C2  If  the  program’s  training  plan  relies  on 
the  organization,  an  organizational 
training  plan  exists  that  identifies: 

•  how  organizational  training  needs 
are  addressed  (formal  courses,  etc.) 

•  the  resources  needed 

•  the  schedule  for  conducting 
training 

•  relationship  of  organizational 
training  to  program  training  needs 

Q2 


Ql  How  are  the  program’s  software 

development  training  needs  planned  and 
implemented?  Identify  the  skill  needs  that 
must  be  addressed.  What  training  vehicles 
will  be  used  to  impart  those  skills?  What 
resources  are  planned  to  develop  those 
skills?  Which  training  vehicles  are  provided 
by  the  program  and  which  are  provided  by 
the  organization?  Does  the  schedule  for 
required  training  meet  program  need  dates 
for  skilled  personnel?  Cl 


Q2  Do  the  program’s  planned  training  needs 
rely  on  the  organization  for 
implementation?  If  so,  describe  what  the 
organization’s  training  plan  covers.  What 
organizational  training  needs  are  addressed? 
Which  skill  needs  of  the  program  are 
addressed?  How  does  the  training  schedule 
reflect  when  those  skills  are  needed  by  the 
program?  What  training  vehicles  will  be 
used  to  impart  those  skills?  What  resources 
are  planned  to  develop  those  skills?  C2 
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Organizational  Resources  and  Program  Support 
Training 

Training  Records  and  Effectiveness 


A  procedure  is  established  and  used  to 
determine  whether  individuals  have  the 
knowledge  and  skills  required  to  perform 
in  their  designated  roles  and  to  document 
training  courses  taken  by  individuals  in 
their  records.  Q1 

Course  effectiveness  is  evaluated  to  help 
ensure  that  the  training  courses  provide 
the  required  training.  Q2 


The  organization  provides  software 
training  and  motivation  and  incentives 
for  personnel  to  take  the  training.  Q3 


Describe  the  process  to  determine  whether 
individuals  have  the  software  development 
knowledge  and  skills  required  to  perform  in 
their  designated  roles.  Where  skills  are 
required  to  meet  program  needs,  how  are 
these  skills  developed?  Cl 

Describe  how  the  effectiveness  of  courses 
are  evaluated.  How  are  the  results  of  the 
evaluations  used  to  revise  the  training  to 
better  meet  the  specific  needs  they  were 
intended  to  address?  C2 

What  courses  are  provided  by  the 
organization  and  how  are  personnel 
encouraged  to  take  the  courses?  C3 


Organizational  Resources  and  Program  Support 
Training 

Training  Requirements 


Cl  Technical  and  management  skills  training  Q1 
is  provided  for  software  development, 
including: 

•  software  engineering  development 

•  the  programming  languages  (e.g., 

Ada) 

•  software  engineering  development 
environments;  e.g.,  System/Software 
Engineering  Environment  (S/SEE) 

•  methods  and  tools 

•  software  project  management 

C2  All  new  software  development  team  Q2 

members  receive  training  in  the 
program’s  software  processes.  Q2 


All  new  software  project  managers  and 
program  managers  receive  training  or 
orientation,  as  appropriate,  in  such  areas 
as  software  project  management,  program 
management,  and  system  engineering. 

Q3 


Identify  the  skills  training  courses  typically 
provided  for  software  development,  in 
particular,  software  engineering 
development,  programming  languages,  and 
software  project  management  development 
environments  (e.g.,  S/SEE),  methods  and 
tools,  and  software  project  management. 
What  program  skill  needs  are  addressed? 
Does  the  program  depend  on  the 
organization  to  provide  any  of  this  training? 
Cl 


Is  training  provided  in  the  program’s 
system  development  and  software 
development  process  and  tools  (e.g., 
S/SEE)?  If  such  training  is  to  be  provided, 
what  topics  are  covered?  C2 

What  training  and  orientation  is  provided  to 
new  software  engineers,  project  managers, 
and  program  managers?  Does  the  program 
depend  on  the  organization  to  provide  any 
of  this  training?  C3 
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5  Organizational  Resources  and  Program  Support 

5.4  Human  Resources 

5.4.1  Manpower  Allocation  Process 


Cl  A  process  exists  for  sizing  the  software 
development  manpower  requirement  and 
for  realistically  allocating  and 
distributing  this  manpower  over  the 
development  phases.  This  process  covers 
both  technical  and  management 
manpower,  is  based  on  a  documented 
model,  and  is  calibrated  on  the  basis  of 
actual  experience.  The  estimating  and 
allocation  process  covers  incremental 
software  development.  Q1 

C2  The  educational  and  training  background  Q2 
of  the  proposed  software  development 
personnel  is  consistent  with  the 
program’s  skill  needs.  Q2 

C3  The  staff  assigned  to  the  subject  program  Q3 
have  the  qualifications,  technical  skills, 
and  experience  in  the  application 
domains  relevant  to  this  program.  Q2 

Q3  Q4  Q5  Q6  Q7 


Q1  Describe  the  manpower  profile  used  to 

allocate  the  software  development  personnel 
over  the  total  program  development  period, 
from  requirements  definition  to 
subsystem/system  testing,  and  explain  the 
basis  for  this  profile.  Cl 


Describe  the  academic  requirements  and 
standards  for  software  development 
engineers.  C2  C3 

What  percentage  of  your  software 
development  personnel  have  science, 
engineering,  mathematics,  or  computer 
science  degrees?  What  percentage  of  the 
people  assigned  to  the  subject  program  will 
have  this  educational  background?  C3 


C4  The  lead  software  development  engineers 
and  managers  assigned  to  the  subject 
program  have  successfully  demonstrated 
technical  leadership  and  management 
skills  on  similar  programs.  Q8  Q9 


Q4  Discuss  the  experience  base  and  numbers  of 
personnel  required  to  accomplish  the 
software  development  and  related  systems 
engineering  and  testing  on  this  program. 

C3 

Q5  Identify  the  average  number  of  years  of 
relevant  software  development  experience 
among  your  total  software  development 
staff.  C3 


Q6  What  percentage  of  this  experience  was 

acquired  while  with  the  current  employer? 
C3 

Q7  Describe  the  software  development 
experience  of  your  staff  in  terms  of 
applications  (domains)  relevant  to  the 
subject  program.  C3 

Q8  Describe  the  software  management 

experience  of  your  software  management 
staff  in  terms  of  applications  (domains) 
relevant  to  the  subject  program.  C4 

Q9  Describe  instances  from  similar  efforts 

where  lead  software  development  engineers 
and  managers  assigned  to  this  program  have 
successfully  demonstrated  technical 
leadership  and  management  skills.  C4 
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5 

5,5 

5.5.1 


Organizational  Resources  and  Program  Support 
Technology  Assessment  and  Transition 
Technology  Transition  Planning 


Cl  Program  needs  are  analyzed  to  identify 
required  capability  areas  that  need  or 
could  benefit  from  new  technology.  Q1 


C2  Responsibilities  are  assigned  and 
resources  allocated  for  monitoring, 
assessing,  selecting,  and  adopting  new 
technologies  for  the  identified  capability 
areas  that  need  or  could  benefit  from  new 
technology.  Q2 


Qi  Describe  your  approach  for  determining  the 
requirements  and  capability  areas  where  new 
technologies  are  needed  or  would  be  most 
helpful.  How  do  you  factor  in  the  ongoing 
and  planned  program’s  needs?  Cl 

Q2  Which  group(s)  or  function(s)  are 

responsible  for  monitoring,  assessing, 
selecting,  and  adopting  new  technologies? 
Do  these  resources  belong  to  the  program 
only  or  to  the  organization?  If  for  the 
organization,  in  what  ways  do  their  activities 
benefit  the  specific  program?  Which 
capability  areas  that  need  or  could  benefit 
from  new  technology  are  targeted  for 
technology  improvement?  What  expertise 
(e.g.,  technology  change  management, 
System/Software  Engineering  Environment, 
measurement)  will  be  available  to  help  in 
assessing  and  transitioning  technology 
changes?  C2 
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5 

5.6 

5.6.1 


Organizational  Resources  and  Program  Support 
Organizational  Process  Management 
Process  Planning  and  Coordination 


Cl  An  organizational  plan  for  improvement 
of  system  and  software  development 
process(es): 

•  is  based  on  action  plans  resulting 
from  assessments  of  the  system  and 
software  development  processes 

•  identifies  highest  priority  areas  for 
improvement 

•  indicates  resources  and  assignments 
to  develop  the  process 
improvements 

•  identifies  applicable  procedures 

•  identifies  how  these  improvements 
are  incorporated  into  ongoing  and 
future  programs 

Ql  Q2 

C2  The  system  and  software  process 

management  activities  of  the  organization 
are  coordinated  (in  particular,  these 
activities): 

•  defining  and  managing  changes  to 
the  organization’s  system  and 
software  processes 

•  collecting  and  maintaining  data  on 
use  of  the  organization’s  system 
and  software  processes 

Q3 


Ql  How  is  the  program  plan  for  system  and 

software  development  process  improvement 
based  on  action  plans  resulting  from  process 
assessments?  Which  processes  are  covered 
in  a  process  assessment?  Are  system 
processes  included?  How  are  findings  from 
the  assessment  typically  addressed  (e.g., 
through  action  plans  which  identify  the 
changes  to  be  made)?  What  are  the  plan’s 
highest  priority  areas  for  improvement? 
What  are  the  program’s  priority  areas  for 
improvement  and  how  are  these  addressed 
in  the  plan?  Cl 


Q2  Which  activities  are  covered  in  the 

organizational  plan  for  system  and  software 
development  process  improvement?  Are 
group  and  individual  responsibilities 
assigned  and  resources  identified?  Identify 
the  procedures  documented  or  referenced  in 
your  plan.  How  are  improvements  to  be 
incorporated  into  ongoing  and  future 
programs?  Cl 


Q3  Which  individual(s)  or  group(s)  are 

responsible  for  coordinating  the  system 
development  and  software  development 
process  management  activities  of  the 
organization?  Who  is  responsible  for 
managing  changes  to  the  organization’s 
system  and  software  development  processes? 
Who  is  responsible  for  collecting  and 
maintaining  data  on  use  of  the 
organization’s  system  and  software 
development  processes  and  making  it 
available  to  other  programs?  How  are  these 
activities  coordinated  with  the  program?  C2 


5 

5.6. 

5.6.2 


Organizational  Resources  and  Progam  Support 
Organizational  Process  Management 
Improvement  Process 


Cl  Specific  opportunities  for  system  and 
software  development  process 
improvement,  addressing  any  area  of 
system  or  software  development  process, 
are  documented  and  can  be  proposed  by 
anyone.  Ql 

C2  Systems  and  software  development 
process  improvement  proposals  are 
evaluated  and  decisions  on  whether  or  not 
to  implement  them  are  made  based  on 
expected  benefits  and  relative  priority. 

Q2 


C3  When  the  decision  is  made  to  transfer  a 
system  or  software  development  process 
improvement  into  a  program,  the 
improvement  is  implemented  in  a  way 
that  ensures: 

•  necessary  resources  to  implement 
the  improvement  are  determined 
and  established 

•  the  appropriate  defined 
development  process(es)  and 
training  courses  are  updated 

•  consultation  support  is  established 

•  changes  in  development  process 
performance  are  measured 

Q3 

C4  Managers  and  technical  staff  are 

informed  of  the  status  and  results  of  the 
organization’s  and  program’s  activities 
for  system  and  software  process 
development  and  improvement.  Q4 


Ql  By  what  mechanism(s)  are  specific 

opportunities  for  process  improvement 
documented  and  submitted?  Which 
employees  can  make  use  of  these 
mechanism(s)?  What  areas  of  the  system 
and  software  development  process  can  they 
address?  Cl 

Q2  Describe  how  employee-identified  and  other 
proposed  opportunities  for  process 
improvement  are  evaluated.  What  criteria 
are  used  to  determine  whether  or  not  to 
implement  a  particular  proposed 
improvement?  How  are  benefits  and 
priorities  of  a  proposed  improvement 
determined?  Which  group(s)  or 
individuals(s)  are  assigned  responsibility  for 
evaluating  and  tracking  these  processes 
improvement  proposals?  C2 

Q3  When  the  decision  is  made  to  transfer  a 
system  or  software  development  process 
improvement  into  the  program,  what  steps 
do  you  take  to  incorporate  the 
improvement?  What  kinds  of  resources  are 
assigned?  How  are  the  applicable  document 
process(es)  (e.g.,  program’s,  organization’s) 
and  training  updated  to  incorporate  the 
improvement?  What  training  and 
consultation  support  do  you  typically  plan 
to  provide?  How  do  you  determine  whether 
the  change  in  process  has  improved 
technical  performance  and  product  and 
determine  cost  benefits?  C3 


Q4  What  groups  and  functions  are  informed  of 
the  status  and  results  of  the  organization’s 
and  program’s  activities  for  system 
development  and  software  development 
process  improvement?  How  are  they 
informed  and  how  often?  C4 


5  Organizational  Resources  and  Program  Support 

5.7  System/Software  Engineering  Environment  (S/SEE) 

5.7.1  S/SEE  Deflnition  Process 


Cl  The  S/SEE  requirements  definition 

process  takes  into  account  the  needs  of 
the  program.  Q1  Q2  Q3 

Qi 

Describe  the  process  by  which  the  S/SEE 
requirements  are  defined  for  the  program. 

Cl 

C2  The  S/SEE  requirements  definition 

process  takes  into  account  past  usage  of 
the  S/SEE.  Q4Q5 

Q2 

How  are  the  tools  selected,  and  what  is  the 
involvement  of  program  personnel  in  the 
selection?  Cl 

Q3 

How  do  you  verify  that  the  program’s  needs 
are  met  by  the  S/SEE?  Cl 

Q4 

How  do  you  leverage  on  lessons  learned 
from  the  use  of  this  S/SEE  on  other 
programs?  C2 

Q5 

Is  there  a  mechanism  for  collecting  S/SEE 
usage  data  and  analyzing  the  effects  of  the 
S/SEE  on  software  quality  and  productivity? 

If  so,  are  the  results  of  this  data  collection 
used  in  the  definition  of  the  program’s 
S/SEE?  C2 
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5 

5.7 

5.7.2 


Organizational  Resources  and  Program  Support 
System/Software  Engineering  Environment  (S/SEE) 
S/SEE  Components 


Cl  The  S/SEE  components  support  the  Q1 

program’s  software  engineering 
development  and  management 
requirements,  functions,  methodologies, 
and  activities.  Q1  Q2  Q3 

C2  The  S/SEE  components  are  mature  and  Q2 
well  documented.  New  tools  are 
determined  through  systematic  evaluation 
to  meet  program  needs.  Q4 


C3  The  selected  compiler  has  been  Q3 

benchmarked  against  program  needs  in 
terms  of  specific  domain  and  application 
requirements  and  functions.  Q5  Q6 

C4  The  S/SEE  components  are  selected  to  Q4 
provide  maximum  commonality  in 
support  of  an  integrated  development 
across  team  members,  whenever 
prime/subcontractor  teaming 
arrangements  exist.  Q7 


Q5 

Q6 


Q7 


Which  software  engineering  development 
and  software  management  requirements 
functions,  methodologies,  and  activities  are 
supported  by  the  S/SEE  and  how?  Cl 

Describe  how  the  hardware  (configuration, 
hosts,  targets,  workstations,  networks,  disks, 
memory  devices,  and  systems,  etc.)  and 
associated  operating  systems  support 
program  needs  in  terms  of  location,  number 
of  users,  volume  of  computation,  and 
compatibility  with  other  contractors.  Cl 

Describe  how  each  tool  in  the  S/SEE 
supports  the  software  development  process 
functions  and  methodologies  selected  for 
the  program.  Cl 

For  each  tool  in  the  S/SEE,  describe  its 
functionality,  its  maturity,  the  quality  of  its 
documentation,  and  how  it  will  be  supported 
during  the  program.  Explain  the  rationale 
for  selecting  new  (not  yet  matured)  tools 
and  how  confidence  is  established  in  the 
ability  of  these  new  tools  to  meet  program 
needs.  Describe  the  S/SEE  as  an  integrated 
whole  and  explain  how  the  S/SEE  will 
efficiently  and  effectively  perform  its 
intended  function.  C2 

Describe  your  efforts  in  testing  the  selected 
implementation  language  compiler(s).  C3 

Describe  your  efforts  in  benchmarking  the 
selected  compiler  relative  to  the  specific 
program  application  and  domain  needs. 
Have  any  guidelines  been  generated  for  use 
of  the  compiler  on  the  program  (special 
language  features,  run-time  system  interface, 
interface  with  other  languages,  etc.)?  C3 

Are  all  components  of  the  S/SEE  common 
across  all  members  of  the  bidding  team? 
Identify  those  that  are  not  and  provide  a 
rationale  for  selecting  them.  C4 
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5  Organizational  Resources  and  Program  Support 

5.7  System/Software  Engineering  Environment  (S/SEE) 

5.7.3  S/SEE  Architecture 


Ql  To  what  level  are  the  components  of  the 
S/SEE  integrated?  Cl 

Q2  Is  there  a  common  user  interface  to  all  the 
services  provided  by  the  S/SEE?  Cl 

Q3  Describe  the  mechanism  for  adding  new 
tools  to  the  S/SEE.  Cl 

Q4  What  indicators  are  used  to  monitor 

memory  utilization  and  throughput  of  the 
S/SEE?  C2 

Q5  Describe  the  process  for  regenerating  each 
software  product  generated  with  the  S/SEE 
(new  software  build,  new  design 
specification,  new  requirements 
specification,  etc.)  after  an  update  has  been 
made.  Estimate  the  time  duration  based  on 
software  size  similar  to  this  program.  C2 

Q6  Describe  the  S/SEE  security  provisions  and 
how  they  are  used  to  manage  unwanted 
intrusions  and  to  protect  information, 
consistent  with  the  program’s  requirements. 
C3 


Cl  The  S/SEE  is  extendible,  easy  to  use,  and 
well  integrated.  Ql  Q2  Q3 

C2  The  S/SEE  memory  utilization  and 

throughput  meet  program  requirements. 

Q4  Q5 

C3  The  S/SEE  security  capabilities  meet 
program  requirements.  Q6 
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5  Organizational  Resources  and  Program  Support 

5.7  System/Software  Engineering  Environment  (S/SEE) 

5.7.4  S/SEE  Maintenance  and  User  Support 

Cl  A  process  exists  to  ensure  that  S/SEE  Q1  How  are  problems  with  the  S/SEE  reported 

problems  are  identified  and  corrected,  and  corrected?  Cl 

and  that  changes  to  the  S/SEE  do  not 
adversely  impact  the  program.  Q1  Q2 
Q3 

C2  A  process  exists  to  ensure  that  the  Q2  How  are  changes  to  the  S/SEE  managed  and 

program’s  S/SEE  users  are  adequately  controlled?  Cl 

trained  and  supported.  Q4  Q5 

Q3  Describe  the  process  of  determining  what 
portions  of  the  S/SEE  must  be  recompiled 
when  a  component  of  the  S/SEE  is 
modified.  When  a  change  is  made  to  the 
S/SEE,  how  is  the  potential  impact  on  the 
program  determined?  Cl 

Q4  Who  has  the  responsibility  to  support  the 

program’s  S/SEE  users  in  their  daily  use  of 
the  S/SEE?  How  is  this  support  managed? 
C2 

Q5  Are  there  training  courses  on  how  to  use  the 
S/SEE?  C2 
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Organizational  Resources  and  Program  Support 

System/Software  Engineering  Environment  (S/SEE) 

Deliverable  S/SEE 

Cl 

The  S/SEE  needed  in  the  field  to  support 
all  of  the  deliverables  has  been  identified. 

Qi 

Ql 

What  S/SEE  services  will  be  needed  in  the 
support  phase  of  the  program?  Cl 

C2 

The  installation  and  support  of  the 
deliverable  S/SEE  has  been  planned.  Q2 

Q2 

What  resources  will  be  needed  for  installing 
the  deliverable  S/SEE?  Have  they  been 
planned?  C2 

C3 

All  restrictions  on  the  use  of  the  S/SEE  or 
its  components  have  been  identified,  and 
are  consistent  with  the  life-cycle  support 
requirements  identified  in  the  program’s 
Request  for  Proposal.  Q3  Q4 

Q3 

Describe  your  plans  for  supporting  the 
deliverable  S/SEE.  C3 

Q4 

Are  there  any  restricted  rights,  licensing,  or 
other  restrictions  on  delivering  any 
components  of  the  S/SEE?  C3 
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Functional  Area  6:  Program  Specific  Technologies 
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6 

6.1 

6.1.1 

Program  Specific  Technologies 
Artiflcial  Intelligence  (AI) 

AI  Task  Domain  Analysis 

Cl 

The  contractor  demonstrates  an 
understanding  of  the  requirements  it  is 
proposing  to  implement  with  AI 
technology.  Q1 

Q1 

Characterize  the  problem  that  the  AI 
implementation  (system)  addresses.  Cl 

C2 

The  contractor  demonstrates  an 
understanding  of  the  functional 
characteristics  of  the  problem  to  which  AI 
technology  will  be  applied.  Q2 

Q2 

What  is  the  input/output  behavior  of  the 
system?  Can  a  sample  dialog/script/trace  be 
provided?  C2 

C3 

The  contractor  understands  the  time  and 
space  constraints  under  which  the  AI 
implementation  will  operate  (including 
development  and  operational  constraints). 

Q3 

Q3 

What  are  the  operational  time  and  space 
constraints  for  the  proposed  system?  C3 

C4 

The  contractor  is  familiar  with  state-of- 
the-art  literature  on  similar  systems.  Q4 

Q4 

Describe  the  current  academic,  industrial, 
commercial,  and  Government  approaches  to 
implementing  similar  systems.  C4 

C5 

The  contractor  has  past  experience  with 

AI  solutions  of  the  class  they  propose. 

Q5 

Q5 

Describe  any  previous  experience  with 
implementing  AI  solutions  of  the  class 
proposed.  C5 
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6 

6.1 

6.1.2 


Program  Specific  Technologies 
Artificial  Intelligence  (AI) 

AI  Tools  and  Technology 


Cl  The  contractor  demonstrates  an 

understanding  of  the  technology  they  are 
proposing,  competing  technologies,  and 
justification  of  their  chosen  technology. 
Q1  Q2  Q3 

C2  The  contractor  proposes  and  has  the  tools 
available  to  perform  statistical  analysis  of 
the  system  behavior  (e.g.,  number  of  rule 
firings,  rule  competitions,  length  of 
reasoning  chains)  or  identifies  plans  for 
acquiring  the  tools.  Q4 

C3  The  contractor  has  experience  with  any 
proposed  commercial  off-the-shelf 
(COTS)  tools,  or  access  to  experience. 

Q5  Q6 

C4  The  contractor  demonstrates  an 

understanding  of  the  canonical  form  of 
knowledge  in  the  system  (e.g.,  rules, 
frames,  networks),  and  employs 
taxonomy  for  the  types  of  knowledge 
used  by  the  system.  Q7  Q8  Q9 

C5  A  process  exists  for  analysis  of  system 
behavior  and  verification  and  validation 
of  the  operational  implementation.  QIO 

Qll 

C6  The  contractor  has  identified  all  of  the 
tools  necessary  to  develop  the  AI 
software.  Q12  Q13  Q15  Q16 

C7  All  of  the  tools  exist  and  have  been 
successfully  used  by  the  offeror.  Q14 

Q15  Q16 


Q1  What  are  competing  approaches  to 
implementing  the  AI  solution?  Cl 


Q2  Why  is  the  proposed  approach  superior  to 
competing  approaches?  Cl 


Q3  What  are  the  limitations  of  the  proposed 

approach?  Are  there  special  circumstances 
for  which  the  proposed  approach  will  not 
work?  Cl 

Q4  Describe  the  reporting  facilities  of  the 

proposed  statistical  analysis  tools,  the  type 
of  statistical  data  that  the  tools  produce,  and 
how  that  information  will  be  used  during 
development.  C2 

Q5  Describe  any  previous  experience  with  the 
proposed  COTS  products.  In  the  absence  of 
previous  experience,  provide  the  sources  of 
expertise  that  will  be  applied  to  this 
program.  C3 

Q6  Why  were  the  particular  COTS  products 

selected?  What  other  COTS  products  exist 
for  the  task,  and  how  are  the  selected 
products  superior?  C3 

Q7  What  is  the  form  of  knowledge  in  the 

proposed  AI  implementation?  Is  there  a 
taxonomy  of  knowledge  classes  that  the 
system  uses?  C4 

Q8  Describe  the  knowledge  engineering 
approach  for  the  proposed  AI 
implementation.  Describe  any  previous 
experience  in  developing  a  knowledge  base 
using  this  engineering  approach.  C4 

Q9  What  is  the  form  of  knowledge  selected  and 
how  is  it  superior  to  other  knowledge 
representation  formalisms?  C4 

QIO  Describe  the  best  and  worst  case 

performance  expectations  for  the  proposed 
AI  implementation.  C5 
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6  Program  Specific  Technologies 

6.1  Artificial  Intelligence  (AI) 

6.1.2  AI  Tools  and  Technology  (Cont’d) 

Qll  Describe  methods  for  verification  and 
validation  of  the  operational 
implementation.  C5 

Q12  Identify  the  tools  that  comprise  the 
proposed  AI  tool  set.  Describe  the 
established  tool  set  selection  criteria.  C6 

Q13  Is  this  tool  set  complete  relative  to 
supporting  the  subject  AI  software 
development?  If  not,  what  tools  are 
missing?  How  will  these  be  acquired  in  time 
to  support  the  development?  C6 

Q14  Describe  any  experience  with  the  proposed 
tool  set.  Also  describe  the  level  of  maturity 
of  the  proposed  tool  set.  C7 

Q15  Identify  any  limitations  of  the  tool  set  and 
plans  to  work  around  these  limitations.  C6 

C7 

Q16  Identify  any  program-specific 

enhancements  planned  for  any  of  the  tools. 

C6  C7 
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6  Program  Specific  Technologies 

6.1  Artificial  Intelligence  (AI) 

6.1.3  Specific  AI  Technology 

Cl  (Expert  Systems)  The  contractor  has  Q1  Describe  the  sources  and  methods  used  to 

access  to  experts  from  which  the  acquire  expert  knowledge.  Cl 

expertise  can  be  extracted.  Q1  Q2 

C2  (Rule-based  System)  The  contractor  Q2  Describe  methods  for  modifying  the 

demonstrates  an  understanding  of  why  a  existing  knowledge  base  as  new  knowledge 

rule-based  approach  is  suitable  for  its  becomes  available.  Cl 

problem  (as  opposed  to  an  algorithmic 
approach).  Q3  Q4 

C3  (Off-line  Training)  The  contractor  can  Q3  Why  is  a  rule-based  approach  better  than  an 

provide  sufficient  training  data  algorithmic  approach?  C2 

(including  the  source  of  the  data  and 
how  well  the  data  mirrors  the 
distribution  of  operational  input).  Q5 
Q6 

C4  (Neural  Networks)  The  contractor  can  Q4  What  are  the  projected  cost  savings  for  using 

identify  the  type  of  learning  that  the  a  rule-based  approach  over  conventional 

network  will  perform,  and  the  equations  programming  techniques?  C2 

that  will  be  used  by  the  connectionist 
units.  Q7 

C5  (Neural  Networks)  The  contractor  can  Q5  What  is  the  source  of  training  data?  How 

identify  the  encoding  of  input  and  does  it  correspond  to  input  that  the  system 

output  data  on  feature  vectors.  Q8  will  see  operationally?  How  is  it  determined 

that  the  system  will  not  overtrain  on  the 
training  data?  C3 

C6  (Genetic  Algorithms)  The  contractor  has  Q6  Describe  the  procedure  for  generating 
defined  the  evaluation  procedures  and  training  data.  Describe  the  process  for 

how  generations  are  constructed.  Q9  establishing  the  criteria  for  how  well  the 

system  performs  on  testing  data.  Relate 
these  criteria  to  operational  usefulness.  C3 

C7  (Genetic  Algorithms)  The  contractor  has  Q7  Describe  the  type  of  connectionist  network 

defined  how  the  evaluation  function  proposed,  and  the  equations  for  weight 

relates  to  operational  performance.  QIO  modification  and  node  firing.  C4 

C8  (Machine  Learning  Systems)  The  Q8  How  is  input  and  output  data  encoded  for 

contractor  demonstrates  an  use  by  the  network?  C5 

understanding  of  when  and  what  the 
system  learns,  and  how  the  learned 
information  is  evaluated  and  used.  Qll 

C9  (Search  Algorithms)  The  contractor  Q9  What  percentage  of  the  population  is  used  to 

demonstrates  an  understanding  of  the  construct  the  next  generation?  What 

time  and  space  requirements  of  the  mutation  functions  are  used,  and  what 

search  procedure.  Q12  percentage  of  a  population  is  mutated? 

Describe  the  evaluation  function,  including 
how  the  genome  is  encoded.  C6 
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6  Program  Specific  Technologies 

6.1  Artificial  Intelligence  (AI) 

6.1.3  SpeciHc  AI  Technology  (Cont’d) 


CIO  (State  Space  Search)  The  contractor 
demonstrates  an  understanding  of  the 
state  space  representation  of  the  problem 
space.  Q13 

Cll  (Heuristic  Search)  The  contractor  has 
demonstrated  that  their  proposed  search 
technique  is  admissible  (always  finds  an 
answer),  or  optimal  (always  finds  the 
best  answer).  Q14 

C12  (Case-Based  Reasoning)  The  contractor 
demonstrates  an  understanding  of  the 
representation  of  a  case,  the  indexing 
scheme  for  cases,  and  how  cases  are 
compared,  modified,  and  evaluated. 

Q15 

C13  (Model-Based  Reasoning)  The 
contractor  demonstrates  an 
understanding  of  the  representation  of  a 
model,  and  how  the  model  is  used  to 
evaluate  assertions  of  system 
performance.  Q16 

C14  (Logic)  The  contractor  demonstrates  an 
understanding  of  the  type  of  logic  being 
proposed,  and  the  proof  procedure  for 
formulas.  Q17 

CIS  (Logic)  The  contractor  demonstrates  an 
understanding  of  the  axioms  used  by  the 
logic.  Q18 

C16  (Fuzzy  Logic)  The  contractor 

demonstrates  an  understanding  of  the 
motivation  and  advantages  of  using 
fuzzy  logic,  and  the  overhead  involved. 

Q19 


QIO  How  does  the  evaluation  function  relate  to 
operational  performance  (i.e.,  if  the 
evaluation  is  .9,  does  that  mean  that  the 
system  will  work  90%  of  the  time?)  C7 

Qll  What  new  knowledge  does  the  system 

acquire?  Describe  the  learning  method.  Is 
learning  done  off-line  or  operationally? 
How  does  the  system  validate  learned 
knowledge?  C8 

Q12  What  are  the  time  and  space  characteristics 
of  the  search  procedure?  C9 


Q13  What  is  the  representation  of  a  node  in  the 
problem  space?  What  are  the 
operations/moves/arcs  to  move  through  the 
state  space?  Is  the  state  space  a  tree  or  a 
graph?  CIO 


Q14  Is  the  search  algorithm  admissible? 
Optimal?  Cll 


Q15  What  is  the  representation  of  a  case?  How 
are  cases  indexed,  compared,  modified,  and 
evaluated?  C12 

Q16  How  is  the  model  represented?  What 
methods  are  used  to  evaluate  model 
performance?  C13 


Q17  What  type  of  logic  is  being  proposed,  and 
what  is  the  proof  procedure  used  to  prove 
formulas?  C14 

Q18  What  are  the  axioms?  What  theorems  have 
been  proved  about  the  axiom  set?  CIS 

Q19  What  advantages  does  fuzzy  logic  have  over 
conventional  logic  for  the  proposed 
solution?  What  overhead  does  the 
calculation  of  fuzzy  values  add?  What 
method  is  used  for  propagating  values? 

C16 
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6 

6.1 

6.1.4 


Program  Specific  Technologies 
Artificial  Intelligence  (AI) 

AI  Management  Process 


Cl  The  contractor  has  a  documented  process 
for  the  engineering  management  and 
development  of  AI  software  and  systems. 

Q1  Q2  Q3 

C2  The  management  process  includes 

statusing  and  controlling  mechanisms, 
with  objective  measures.  Q4  Q5 

C3  A  process  exists  to  establish  schedule 
durations,  milestones,  and  effort 
allocation  for  the  AI  development  effort. 
Q6  Q7 

C4  The  contractor  has  identified  any  unique 
work  package  requirements  driven  by  AI 
development  management  requirements. 
Q8  Q9 

C5  The  contractor  has  a  process  for 

estimating  software  size  which  is  based  on 
actual  AI  software  development 
experience.  QIO  Qll 

C6  The  AI  technology  necessary  to  develop 
the  subject  program  exists  and  has  been 
successfully  applied  by  the  contractor. 

Q12  Q13 


Q1  Describe  the  process  for  managing  the  AI 
development  effort.  Cl 


Q2  What  specific  process  steps  are  used  to 
manage  the  AI  software  development 
process?  What  is  the  role  of  systems 
engineering?  Cl 

Q3  Has  the  standard  software  development 

management  process  for  AI  been  adapted? 
Describe  these  adaptations.  Cl 

Q4  Describe  the  specific  mechanisms  proposed 
to  status  and  control  AI  software 
development  over  the  system  development 
period.  C2 

Q5  Describe  how  the  AI  development  progress 
will  be  measured  and  assessed  to  include 
analysis,  requirements  definition,  design, 
code  implementation,  integration,  and  test. 
C2 

Q6  How  are  schedule  durations  of  the  AI 
development  and  its  impact  on  the  total 
program  schedule  determined?  C3 

Q7  Describe  the  milestones,  periods,  and 
software  effort  allocated  over  the  AI 
software  development  schedule.  C3 

Q8  Has  the  definition  and  implementation  of 
work  packages  for  the  AI  software 
development  effort  been  modified 
compared  to  the  standard  process?  If  so, 
describe  the  work  packages  intended  to  be 
used  to  plan,  define,  control,  and  status  the 
development  effort  and  why  they  have  been 
modified.  C4 

Q9  Identify  the  milestone  product  completion 
criteria  for  the  various  phased  products 
relative  to  work  completion.  C4 

QIO  Describe  the  management  approach  to 
estimating  the  size  of  the  AI  software 
development  effort.  Identify  any 
experience  base  used  for  this  estimate. 
Describe  how  the  estimating  process  reflects 
actual  completed  AI  software  development 
efforts.  C5 
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6 

6.1 

6.1.4 


Program  Specific  Technologies 
Artificial  Intelligence  (AI) 

AI  Management  Process  (Cont’d) 


Qll  Recognizing  that  AI  program  size  is  not 
reflected  in  the  empirically  derived 
estimating  models,  how  are  estimates 
established  for  each  of  the  items  below? 

•  Program  size 

•  Effort  required 

•  Development  schedules 

•  Distribution  of  effort  over  the 
schedule 

•  Cost 
C5 

Q12  Identify  any  management  concerns  with  the 
status  of  AI  technology  relative  to 
developing  software  within  the  subject 
program  baseline.  C6 

Q13  What  are  the  specific  strategies  to  manage 
the  risk  associated  with  AI  technology?  C6 
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6  Program  Specific  Technologies 

6.1  Artificial  Intelligence  (AI) 

6.1.5  AI  Development  Process 


Cl 

The  contractor  has  an  engineering 
process  for  AI  software  development  that 
is  documented  and  has  been  successfully 
applied  on  past  AI  development 
programs.  Q1  Q2  Q3 

Ql 

Describe  the  overall  AI  development  ap¬ 
proach.  Identify  each  step  in  the  process, 
and  the  products  of  each  step.  Cl  C2 

C2 

The  engineering  process  for  AI  software 
development  includes: 

•  a  systems  engineering  top-level 
architectural/design  phase 

Q2 

Identify  any  changes  to  the  traditional 
software  development  process  as  a  result  of 
using  AI  technology.  Cl 

•  measurable  milestones  with 
completion  criteria 

•  documentation  of  intermediate 
steps  and  final  product  design 
disclosure 

•  analysis,  requirements  definition, 
design,  code,  test,  and  integration 

•  capturing  and  retaining  the 
rationale  behind  AI  design 
decisions 

Q1  Q4  Q5  Q6 

C3 

The  AI  software/system  development 
process  includes  a  comprehensive 
verification  methodology  and  phase  to 
validate  that  (e.g.),  the  expert  system 
meets  the  specified  performance.  Q7 

Q3 

Has  the  development  process  been  verified 
through  previous  development,  prototype 
development,  or  Independent  Research  and 
Development?  Cl 

Q4 

Describe  how  the  rationale  behind  the  AI 
design  decisions  will  be  captured  and 
retained.  C2 

Q5 

Identify  any  special  test  levels  unique  to  the 

AI  applications.  C2 

Q6 

Describe  any  special  test  facilities  and 
resources  required  that  are  unique  to  AI 
applications.  C2 

Q7 

How  does  the  AI  software/system 
development  process  validate  AI  system 
performance?  C3 
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6 

6.1 

6.1.6 


Program  Speciflc  Technologies 

Artiflcial  Intelligence  (AI) 

Personnel  Skills  and  Qualifications  for  AI 


Cl 

The  contractor  has  the  necessary  AI  skills 
and  experience  to  accomplish  the  AI 
software  and  system  development.  Q1 

Q2  Q3  Q4  Q5 

Ql 

Identify  any  staff  AI  development  skills. 
Discuss  all  necessary  AI  skills  to  execute  the 
subject  AI  development  effort.  Cl 

C2 

The  contractor’s  AI  skills  and  experience 
are  relevant  to  the  subject  program 
application.  Q6 

Q2 

How  were  these  AI  skills  acquired?  Cl 

C3 

A  comprehensive  AI  training  program 
exists  that  is  sufficient  to  develop  and 
maintain  the  skilled  personnel  for  the 
subject  program.  Q7  Q8  Q9  QIO  Qll 

Q3 

How  is  AI  proficiency  measured  and 
evaluated  in  the  various  skills  required?  Cl 

C4 

The  contractor  has  the  skilled  and 
experienced  personnel  available  to 
perform  development  within  the  subject 
program  baselines.  Q12  Q13  Q14 

Q4 

Describe  the  corporate/division  experience 
with  actual  application  of  AI.  Cl 

Q5 

Describe  any  experience  with  the  defined 
development  activities  and  phases.  Cl 

Q6 

Explain  why  the  referenced  AI  experience  is 
relevant  and  provides  a  basis  for  the  subject 
program  development.  C2 

Q7 

Describe  the  training  program  followed  to 
train  personnel  in  AI.  C3 

Q8 

Identify  the  total  length  of  the  training 
period  and  the  subjects  covered.  C3 

Q9 

Does  the  training  provide  technical  and 
management  coverage  ?  Explain.  C3 

QIO 

Is  AI  training  required  for  all  members  of 
the  staff?  C3 

Qll 

How  is  proficiency  developed  following  the 
initial  training?  C3 

Q12 

Demonstrate  that  sufficient  Al-trained  and 
-proficient  personnel  are  available.  How 
many  are  required  throughout  the 
development?  C4 

Q13 

From  where  are  these  personnel  coming? 

C4 

Q14 

What  contingency  provisions  exist  if  enough 
personnel  are  not  available?  C4 
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6  Program  Specific  Technologies 

6.1  Artificial  Intelligence  (AI) 

6.1.7  AI  Capability  Demonstrations  and  Risk  Management 


Cl  The  contractor  has  planned  specific 
demonstrations  to  establish  that  AI 
technology  and  related  tools  exist  in  a 
form  sufficiently  mature  and  dependable 
to  perform  the  subject  program  AI 
software  development.  Q1  Q2 

Q1 

Describe  planned  demonstrations  to 
establish  the  AI  tools,  capability  and 
approach  to  this  development  program.  Cl 

C2  The  contractor  has  documented  risk 
management  methods  to  ensure  a 
successful  development  effort  within  the 
subject  program  baselines.  Q3  Q4  Q5 

Q6  Q7 

Q2 

What  are  the  schedules  and  criteria  for  these 
demonstrations?  Cl 

Q3 

Describe  any  shortfalls  or  deficiencies  seen 
in  the  AI  technology  base  to  support  the 
development  effort.  C2 

Q4 

What  backups,  provisions,  and  contingency 
plans  exist  to  compensate  for  these 
shortfalls?  C2 

Q5 

Recognizing  AI  as  an  evolving  technology, 
what  risks  exist  in  developing  the  subject 
program  using  AI  within  program  con¬ 
straints?  Identify  specific  technical  and 
management  risks.  C2 

Q6 

Describe  any  defined  specific  risk 
management  provisions  planned  for  use. 

C2 

Q7 

Describe  the  criteria  used  to  exercise  each 
risk  management  provision.  C2 
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6  Program  Specific  Technologies 

6.2  Safety-Critical  Digital  Systems  (SCDS) 

6.2.1  Safety-Critical  Program  Management 


Cl  An  approach  is  defined  to  account  for 
changes  to  Government-furnished 
equipment  and  contractor-furnished 
equipment  safety-critical  items  that 
impact  development  efforts.  Q1 

C2  Interfaces  and  management  agreements 
are  proposed  to  facilitate  communication 
and  interaction  of  the  development 
organization  and  other  safety-critical 
developers/maintainers,  other  nonsafety- 
critical  developer/maintainers,  the 
subsystem  Integrated  Product  Team  (IPT) 
lead  management  functions,  and  the 
organization  (IPT)  responsible  for 
management  at  the  system  level  (e.g., 
aircraft  level).  Q2 

C3  The  safety  certification  process  is  defined 
in  the  Software  Development  Plan  (SDP) 
or  other  appropriate  vehicles.  Q3 

C4  The  proposed  schedule  estimation  and 
definition  system  accommodates  the 
safety-critical  development  activities 
defined  by  the  contractor.  Q4 


Qi  Describe  the  proposed  approach  to 

accounting  for  changes  to  Government- 
furnished  equipment  and  contractor- 
furnished  equipment  safety-critical  items 
that  impact  development  efforts.  Cl 

Q2  What  formal  or  informal  agreements  exist 
between  the  development  organization  and 
other  safety-critical  developers/maintainers, 
other  nonsafety-critical 
developer/maintainers,  the  subsystem  IPT 
lead  management  functions,  and  the 
organization  (IPT)  responsible  for 
management  at  the  system  level  (e.g., 
aircraft  level)?  C2 


Q3  Is  the  safety  certification  process  defined  in 
the  SDP?  If  not,  describe  the  vehicle  used  to 
define  the  certification  process.  C3 

Q4  How  is  it  ensured  that  the  schedule  will 
accommodate  all  safety-critical  activities 
required?  C4 


128 


129 


130 


6  Program  Specific  Technologies 

6.2  Safety-Critical  Digital  Systems  (SCDS) 

6.2.2  Safety-Critical  Systems  Engineering  (Cont’d) 


Q12  What  specific  process  and  procedures  are 
used  to  verify  that  the  top-level 
system/subsystem  architecture  meets  the 
system-level  safety-critical  requirements 
(e.g.,  fault  identification)  fault  tolerance? 
How  does  this  process  ensure  flight-critical 
systems  are  designed  with  the  necessary  re¬ 
dundancy  management  to  accommodate 
fault  tolerant  reliability  specific 
requirements?  C8 

Q13  Describe  how  the  top-level  design  approach 
ensures  the  autonomy  and  integrity  of 
SCDS.  How  does  this  process  identify 
safety-critical  component  interconnectivity 
as  it  relates  to  the  identification  and  control 
of  error  propagation  through  the  system? 
How  does  this  process  prevent 
contamination  of  safety-critical  systems  and 
components  by  nonsafety-critical  systems? 
C9 
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6  Program  Specific  Technologies 

6.2  Safety-Critical  Digital  Systems  (SCDS) 

6.2.3  Safety-Critical  Software  Engineering 


Cl  A  process  is  proposed  and  defined  to 
translate  and  trace  safety-specific  system 
requirements  into  the  software 
requirements  baseline.  Q1 


C2  The  identified  top-level  design  process 
defines  the  method  to  verify  the  design 
against  the  established  safety 
requirements,  products,  and  completion 
criteria.  Q2 

C3  A  process  exists  to  evaluate  and  analyze 
the  software  design  to  hazardous 
conditions.  Q3 

C4  A  process  exists  that  continues  the 

identification  of  safety-critical  elements 
down  to  Computer  Software  Components 
(CSCs)  and  Computer  Software  Units 
(CSUs).  Q4 

C5  A  process  is  proposed  that  ensures  the 
autonomy  and  integrity  of  the  structure 
and  interfaces  between  safety-critical 
elements  and  nonsafety-critical  elements, 
configuration  items  (CIs),  Computer 
Software  Configuration  Items  (CSCIs), 
CSCs,  and  CSUs.  Q5  Q6 

C6  A  process  exists  to  trace  safety-critical 
parameters  within  the  code  to  the 
functions  that  modify  them  or  to  the 
functions  that  use  these  safety-critical 
parameters.  Q7 

C7  If  specific  safety-critical  coding  standards 
are  used,  these  standards  are  defined  and 
verified  as  to  compliance.  Q8 


Qi  Describe  the  specific  process  to  translate  and 
trace  safety-specific  system  requirements 
into  the  software  requirements  baseline. 
Describe  how  traceability  of  these  software 
safety-critical  requirements  back  to  their 
higher-level  system-  and  subsystem-level 
safety-critical  requirements  is  maintained. 

Cl 

Q2  What  is/are  the  product(s)  of  the  top-level 
design  activity?  What  are  the  criteria  for 
completion  of  the  top-level  design  activity? 
Describe  the  process  to  assess  whether  the 
software  design  implements  (satisfies)  the 
established  safety  requirements.  C2 

Q3  Describe  the  process  to  evaluate  and  analyze 
software  designs  (top  level  and  detailed 
level)  for  hazardous  conditions.  C3 

Q4  Describe  the  process  to  identify  safety- 
critical  CSCs  and  CSUs.  C4 


Q5  Describe  the  processes  and  procedures  to 

ensure  autonomy  of  structure  and  interfaces 
between  safety-critical  CSCs  and  CIs  and 
nonsafety-critical  CSCs  and  CIs.  C5 


Q6  What  process  is  used  to  ensure  that  design 
changes  to  nonsafety-critical  CSCs  and 
CSUs  do  not  adversely  impact  safety-critical 
CSCs  and  CSUs?  C5 


Q7  Describe  the  approach  to  verifying  that 
safety-critical  parameters  are  properly 
traced  within  the  code  to  the  functions  that 
modify  them  or  to  the  functions  that  use 
these  safety-critical  parameters.  Does  this 
approach  allow  impact  analyses 
performance  to  determine  if  modifications 
made  to  any  part  of  the  system  will  have 
some  effect  on  critical  parameters  contained 
in  safety-critical  software?  Does  this 
approach  facilitate  mapping  safety-related 
message  paths  with  external  interfaces?  C6 
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6 

6.2 

6.2.3 


Program  Specific  Technologies 
Safety-Critical  Digital  Systems  (SCDS) 
Safety-Critical  Software  Engineering  (Cont’d) 


Q8  What  internal  standards  and  procedures 

define  the  safety-critical  coding  standards  to 
be  applied?  Which  organization  verifies 
compliance  to  the  established  coding 
standards?  C7 
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6  Program  Specific  Technologies 

6.2  Safety-Critical  Digital  Systems  (SCDS) 

6.2.4  Safety-Critical  Software  Test  and  Integration  (Cont’d) 


QIO  Describe  the  process  for  determining  the 
required  level  of  test  and  retest  for  safety- 
critical  components.  For  example,  if  an 
error  requiring  a  code  change  is  discovered 
in  subsystem  testing,  describe  the  levels  and 
completeness  of  the  retest  starting  at  the  unit 
level  and  progressing  to  higher  levels.  Cl 
C6 

Qll  How  is  fault  injection  used  in  the  validation 
and  verification  of  safety-critical 
requirements  performance?  What  process  is 
used  to  decide  if  fault  injection  is 
appropriate?  C7 

Q12  Describe  the  use  of  target  digital  processor 
and  other  system  hardware  in  the  successive 
buildup  of  the  software  integration  and  test. 
In  the  absence  of  actual  hardware,  how  are 
software  functions  that  are  dependent  on 
hardware  interfaces  tested?  C8 
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6 

6.2 

6.2.5 


Program  Specific  Technologies 
Safety-Critical  Digital  Systems  (SCDS) 

Safety-Critical  Subsystem/System  Test  and  Integration  (Cont’d) 


C8  Test  cases,  descriptions,  procedures,  and 
reports  are  maintained  and  updated  for 
each  level  to  test  during  subsystem  and 
system  test.  Processes  exist  that  define 
how  the  test  cases,  descriptions, 
procedures  and  reports  are  used  for 
regression  testing.  Q8 

C9  An  approach  to  ensure  adequate 
subsystem  development  team 
involvement  and  support  of  system-level 
integration  and  test  activities  for  SCDS  is 
defined.  This  approach  defines  the 
subsystem  development  team 
involvement  in  the  following  levels  of 
system  integration  and  test: 

•  System  integration  laboratory 
testing 

•  Dynamic  simulation  testing 

•  On-system  ground  test 

•  Flight  tests 

Q9 


CIO  A  procedure  exists  and  is  used  to 
analyze  and  determine  the  level  of 
fidelity  required  at  each  level  of 
subsystem  and  system  test.  QIO 


Q8  Describe  how  subsystem/system  test 

descriptions,  procedures,  and  cases  are 
maintained  during  subsystem  and  system 
test.  How  are  these  procedures  used  for 
regression  testing?  C8 


Q9  Describe  the  approach  to  ensure  adequate 
subsystem  development  team  involvement 
and  support  of  system-level  integration  and 
test  activities  for  SCDS  covering  the 
following  levels:  C9 

•  System  integration  laboratory  testing 

-  Static  environment  testing 

-  Interface  compatibility 

-  Communication/timing 

-  Operability 

•  Dynamic  simulation  testing 

-  Operational  (actual) 
hardware/software 

-  High  fidelity 

environmental/simulation  models 

-  FMET 

-  Performance/operational  confidence 
tests 

-  Interface  compatibility  validation 

•  On-system  ground  test 

-  System 

operational/compatibility/connectvity/ 
integration  tests 

-  Ground  vibration  and  other  ground 
safety  checks 

-  Electromagnetic  countermeasures 
validation 

•  Flight  tests 

-  Controlled  envelope  expansion 

-  Test  start/objectives/criteria  defined 

-  Test  levels/quantities/coverage 
analysis  process 

C9 

QIO  Describe  the  process  to  analyze  and 

determine  the  level  of  fidelity  required  at 
each  level  of  subsystem  and  system  testing. 
Describe  how  the  proposed  facilities  satisfy 
established  fidelity  requirements.  CIO 
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Program  Specific  Technologies 
Safety-Critical  Digital  Systems  (SCDS) 
I  Safety-Critical  Personnel  Resources 


The  application-specific  personnel 
resources  needed  to  support  the  safety- 
critical  design,  development,  integration, 
and  test  effort  are  available  or  a  plan  is 
defined  to  acquire  the  needed  resources. 

Qi 


Are  the  application-specific  personnel 
resources  needed  to  support  the  safety- 
critical  design,  development,  integration,  and 
test  effort  available?  If  not,  describe  plans 
to  acquire  the  needed  resources.  Address 
the  following  combinations  and  types 
depending  on  the  application: 

•  functional  engineers  including  safety 
engineering 

•  systems  engineers  with  specific 
functional  and  integration  experience 
along  with  a  complete  understanding 
of  SCDS  architecture  and  integration 
criticality. 

•  a  strong  understanding  of  how  the 
software  functions  integrate  with  the 
subsystem/system  and  the  software 
development  process  that  must  be 
followed  to  minimize  changes  made  to 
SCDSs. 


The  application-specific  personnel 
resources  available  have  the  relevant 
experience,  academic  qualifications  and 
programming  language  skills  and 
experience  required  for  the  subject 
program.  Q2  Q3  Q4 


Identify  the  average  number  of  years  of 
relevant  safety-critical  system  development 
experience  existing  among  the  safety- 
critical  development  staff  and  contractors. 
C2 


What  are  the  academic  requirements  and 
standards  for  software/system  safety 
engineers  to  be  used  in  the  safety-critical 
software  development?  C2 

Describe  any  experience  with  actual 
application  of  the  selected  programming 
language  to  SCDS  applications.  How  is  this 
experience  relevant  for  this  development? 
C2 
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».3 

i.3.1 


Program  Specific  Technologies 
Complex  Hardware  Development 
Hardware  Management 


Cl  A  process  for  managing  complex  Q1  Describe  the  basic  process  used  to  manage 

integrated  circuit  (CIC)  development  and  CIC  hardware  development.  Cl 

procurement  is  described.  This  process 
includes  the  following: 

•  workload  estimates  and  budgets 

•  development  schedules  and 
relationship  to  specific  work 
packages 

•  methods  for  tracking  progress  of 
individual  work  assignments 

•  work  assignment  prioritization 

•  budget  and  schedule  impacts 

Management  tools  used  in  the 
requirements  flowdown  and  design 
process  are  identified.  Q1  Q2 

C2  A  documented  process  for  managing  the  Q2  How  is  the  workload  estimated  and  bud- 

flowdown  of  CIC  hardware  requirements  geted?  How  are  overall  schedules  developed 

to  individual  integrated  circuits  (ICs)  is  3nd  how  do  they  relate  to  specific  work 

described  03  04  packages  assigned  to  individuals?  How  are 

■  ^  ^  work  assignments  prioritized  and  what 

system  is  used  to  track  progress  of 
individual  work  assignments?  How  are 
budget  and  schedule  impacts  identified 
“from  the  bottom  up”  via  these  detail 
assessments?  Cl 

C3  A  method  to  coordinate  hardware  and  Q3  Identify  any  tools  used  to  assist  in  managing 

software  designs  and  resolve  conflicts  is  of  the  CIC  hardware  requirements  flowdown 

implemented.  Q5  and  design  process.  These  may  include 

computer-based  commercial  off-the-shelf 
packages,  in-house  systems,  or 
nonautomated  accountability  systems.  C2 

Q4  What  is  the  organizational  structure  for 

managing  the  flowdown  of  CIC  hardware 
requirements  to  the  level  of  individual 
complex  ICs?  C2 

Q5  How  will  CIC  hardware  designers  coordinate 
their  designs  with  those  of  software 
designers  via  this  organizational  structure? 
How  are  conflicting  concepts  in  the 
(hardware/software)  designs  surfaced  and 
resolved?  C3 
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.3 

.3.3 


Program  Specific  Technologies 
Complex  Hardware  Development 
Hardware  Design  and  Test 


Cl  A  documented  process  is  described  to 
allocate  design  requirements  to  hardware 
and  software  and  to  identify  specific 
complex  integrated  circuit  (CIC)  types. 

Qi 


C2  Design  validation  is  accomplished  prior 
to  release  to  fabrication.  Q2  Q3 


C3  In-house  standards  for  acceptance  of  CIC 
hardware  have  been  established.  Q4 


Ql  How  are  design  requirements,  once  allocated 
to  hardware  and  software  in  general,  flowed 
down  to  an  identified  need  for  a  particular 
CIC  (e.g.,  an  application-specific  integrated 
circuit  (ASIC)  or  gate  array)?  Describe 
where  this  flowdown  is  documented, 
including  documentation  of  the  specific, 
detailed  requirements  to  be  met  by  the 
identified  CIC.  How  is  this  flowdown 
process  and  documentation  of  the  results 
made  visible/available  to  the  Government? 
Cl 

Q2  Describe  the  process  for  design  validation  of 
CICs  (e.g.,  ASICs,  complex  gate  arrays,  very 
high  speed  integrated  circuits)  prior  to 
release  to  fabrication.  This  description 
should  include  identification  of  design  tools 
and  methodologies,  hardware  and/or 
software  simulators,  development  of  test 
vectors  for  on-board  Built-in  Test  circuits, 
and  any  standards  to  be  met  prior  to  release 
of  the  IC  design.  C2 

Q3  Describe  the  process  to  ensure  that  vendor- 
supplied  chips  are  designed  properly  with 
regard  to  tolerance  buildup,  timing 
variability,  and  other  similar  concerns.  How 
are  these  design  requirements  documented? 
C2 


C4  CIC  documentation  standards  are  Q4  What  standards  exist  for  in-house 

established  consistent  with  the  need  for  acceptance  of  CIC  hardware?  C3 

long-term  support  and  reprocurement  by 
the  program.  Q5 


Q5  Describe  the  documentation  standards  for 
CICs  developed  for  this  project.  How  is 
transportability  of  this  design  information  to 
the  DOD-required  VHSIC  Hardware 
Description  Language  ensured?  How  will 
documentation  produced  through  these 
standards  be  sufficient  to  permit  future 
design  changes  in  IC  functionality  by  other 
than  the  original  developer?  C4 


6  Program  Specific  Technologies 

6.3  Complex  Hardware  Development 

6.3.4  VHSIC  Hardware  Description  Language  (VHDL) 


Cl  The  contractor  has  a  documented 

engineering  process  for  VHDL  model 
development  that  has  been  successfully 
applied  on  previous  hardware 
development  efforts.  Q1  Q2  Q3 

C2  The  contractor  either  acquires  or 

generates  adequate  references  pertaining 
to  the  device  or  system  being  modeled. 

Q4 

C3  The  VHDL  models  conform  to  the  IEEE 
Standard  1076-1987  Language 
Reference  Manual.  Q5 

C4  The  contractor  employs  an  appropriate 
suite  of  VHDL  development  tools.  Q6 
Q7 


C5  The  VHDL  development  methodology 
includes  a  comprehensive  validation 
phase  to  certify  that  the  VHDL  models 
meet  their  specifications.  Q8  Q9 


Ql  Describe  past  experience  with  VHDL  model 
development,  including  the  number  and 
scope  of  VHDL  descriptions  completed.  Cl 


Q2  Describe  the  overall  VHDL  design 

methodology.  Identify  each  step  in  the 
process,  and  the  products  of  each  step. 
Identify  where  this  methodology  is 
documented.  Cl 

Q3  Is  each  physical  implementation  level  of  an 
electronic  system  described  both  structurally 
and  behaviorally?  Cl 

Q4  What  references  determine  the  functionality, 
timing,  and  operation  of  the  electronic 
systems  being  modeled?  Explain  the  typical 
conditions  for  inclusion  of  each  of  the 
following:  military  or  contract 
specifications.  Data  Item  Descriptions 
(DIDs),  system-level  specifications, 
application-specific  integrated  circuit 
(ASIC)  design  specifications,  standard  IC 
databooks/specifications,  and  hardware  test 
plans.  C2 

Q5  How  are  VHDL  models  ascertained  to 

comply  with  IEEE  Standard  1076?  Which 
VHDL  analyzers  are  used  for  compilation? 
C3 


Q6  Describe  the  various  tools  routinely  used  for 
VHDL  development,  covering  analysis, 
simulation,  and  logic  synthesis.  C4 

Q7  Are  the  hardware  designers  trained  in  the 
use  of  VHDL  tools?  C4 

Q8  Describe  the  process  whereby  VHDL 
models  are  tested.  What  constitutes 
specification  of  correct  model  operation? 
Are  all  aspects  of  functional  behavior 
covered  by  a  test  bench?  Are  the  test 
vectors  written  in  the  Waveform  and  Vector 
Exchange  Specification  (WAVES),  IEEE 
Standard  1029.1-1992  format?  C5 

Q9  Are  high-level  simulations  carried  out,  as 
well  as  behavioral  simulations  at  the  card 
level?  How  are  interfaces  between  chips 
simulated?  What  timing  tests  are 
performed?  C5 
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6  Program  Specific  Technologies 

6.4  Database  Management 

6.4.1  Need  for  Database 


Cl  The  contractor’s  processes  and  Q1 

procedures  address  initial  trade-off  issues 
relating  to  database  usage  as  an 
implementing  technology.  Q1  Q2  Q3 

Q4 

Q2 


Q3 


What  issues  have  been  identified  that 
indicate  a  need  to  use  a  database?  Cl 


What  attributes  of  databases  would 
effectively  address  the  issues  identified?  Cl 

In  terms  of  life-cycle  support,  how  does  the 
support  of  a  database  in  the  operational 
mode  compare  to  that  of  the  alternative 
solution?  Cl 


Q4  For  a  distributed  or  heterogeneous 

environment,  what  are  the  critical  technical 
issues  in  database  technology,  and  how  will 
they  be  addressed?  Cl 
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6  Program  Specific  Technologies 

6.4  Database  Management 

6.4.2  Database  Design 


Cl 

The  contractor  demonstrates  an 
understanding  of  how  to  identify  the  best 
database  methodology  to  satisfy  system 
requirements.  Q1  Q2  Q3  Q4 

Qi 

What  is  the  selected  database  model  type? 

Cl 

C2 

The  contractor  demonstrates  an 
understanding  of  how  to  select  an 
acceptable  data  modeling  methodology. 

Q5 

Q2 

What  are  the  competing  approaches  for 
satisfying  the  system  requirements?  Cl 

C3 

A  process  is  defined  for  validating  the 
database  design.  Q6 

Q3 

What  are  the  advantages  of  the  selected 
approach?  Cl 

C4 

The  database  design  is  developed 
interactively  with  the  user.  Q6 

Q4 

What  are  the  limitations  of  the  selected 
approach?  Cl 

Q5 

Describe  the  data  modeling  methodology  to 
be  used.  C2 

Q6 

Describe  the  database  model  type  and 
rationale  for  selection.  How  will  the  chosen 
database  design  be  validated?  How  will  the 
user  be  incorporated  into  the  design 
process?  C3  C4 
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6  Program  Specific  Technologies 

6.4  Database  Management 

6.4.4  Database  Development 


Cl  The  database  development  process  and 
procedures  are  defined  in  internal 
development  standards  and  procedures 
and  include  internal  reviews, 
walkthroughs,  statusing,  testing,  and 
discrepancy  resolution.  Q1  Q2  Q3 

C2  The  database  development  processes  and 
procedures  are  compatible  with  the 
selected  database  methodology.  Q4  Q5 


Qi  Describe  the  database  development 

processes  and  phases  with  respect  to  the 
software  development.  Do  these  phases 
provide  a  realistic  schedule  for  the  database 
development?  Cl 


Q2  Describe  the  internal  review  process  of  the 
database  development  in  terms  of  software 
and  requirement  reviews.  Who  is  involved? 
Who  is  ultimately  responsible?  How  are 
discrepancies  resolved?  Define  the 
progression  of  testing  through  internal 
reviews,  walkthroughs,  and  software 
inspection.  Cl 

Q3  Describe  how  external  reviews, 

walkthroughs,  and  software  inspections  will 
be  carried  out.  Cl 

Q4  Describe  how  the  selected  database 

development  process  is  compatible  with  the 
selected  database  methodology.  C2 

Q5  How  is  data  integrity  enforced  during 
development?  C2 


6  Program  Specific  Technologies 

6.4  Database  Management 

6.4.5  Database  Quality  Assurance 


Cl  A  process  is  defined  for  verification  and 
validation  of  the  database  system.  Q1  Q2 
Q3  Q4  Q5  Q6  Q7 

Q1 

Describe  the  methods  for  verification  and 
validation  of  the  database  system.  Cl 

Q2 

Describe  how  the  software  releases  and 
database  releases  are  configured  with  one 
another.  C 1 

Q3 

How  is  the  interface  between  the  software 
and  the  database  management  system 
tested?  Cl 

Q4 

How  is  a  database  release  version 
documented?  Who  controls  releases?  Cl 

Q5 

How  is  the  integrity  of  the  data  ensured  and 
controlled?  Cl 

Q6 

How  will  spot-checks  of  the  data  by  the 
domain  experts  be  carried  out?  Cl 

Q7 

What  configuration  control  procedures  are 
established  for  the  database?  Cl 
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6  Program  Specific  Technologies 

6.4  Database  Management 

6.4.6  Personnel  Skills  and  Qualifications  for  Database 


Cl  The  contractor  has  the  necessary  database 
skills  and  experience  to  accomplish  the 
database  software  and  system 
development.  Q1  Q2  Q3  Q4  Q5  Q6  Q7 

C2  The  contractor’s  database  skills  and 
experience  are  relevant  to  program 
application.  Q6 

C3  A  comprehensive  database  training 
program  exists  that  is  sufficient  to 
develop  and  maintain  the  skilled 
personnel  for  the  program.  Q7  Q8  Q9 
QIO  Qll 

C4  The  contractor  has  the  skilled  and 
experienced  personnel  available  to 
perform  the  development  within  program 
baselines.  Q12  Q13  Q14 


Q1  Identify  any  staff  database  development 
skills.  Discuss  all  necessary  database  skills 
to  execute  the  subject  database  development 
effort.  Cl 

Q2  How  were  these  skills  acquired?  Cl 


Q3  How  is  database  proficiency  measured  and 
evaluated  in  the  various  skills  required?  Cl 


Q4  Describe  the  corporate/division  experience 
with  actual  application  of  databases  to 
Mission-Critical  Computer  Resources 
applications.  Cl 


Q5  Describe  any  database  experience  pertaining 
to  the  defined  development  activities  and 
phases.  Cl 

Q6  Explain  why  the  referenced  database 

experience  is  relevant  and  provides  a  basis 
to  do  the  subject  program  development.  C2 

Q7  Describe  the  staff  training  program  followed 
to  train  database  development  skills.  C3 

Q8  Identify  the  total  length  of  the  training 
period  and  the  subjects  covered.  C3 

Q9  Does  this  training  provide  technical  and 
management  coverage?  C3 

QIO  Is  database  development  training  required 
for  all  members  of  the  staff?  C3 

Qll  How  is  proficiency  developed  following  the 
initial  training?  C3 

Q12  Demonstrate  that  sufficient  database 
development  trained  and  proficient 
personnel  are  available.  How  many  are 
required  throughout  the  development?  C4 

Q13  From  where  are  these  personnel  coming? 

C4 

Q14  What  contingency  provisions  exist  if  enough 
personnel  are  not  available?  C4 
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6 

6.5 

6.5.1 


Program  Specific  Technologies 

Trusted  Systems 

Trusted  Systems  Engineering 


Cl  The  contractor  understands  how  to 

engineer  security  into  its  system  such  that 
it  is  an  integral  part  of  the  overall  design 
of  the  system.  Q1  Q2  Q3 

C2  The  security  requirements  for  the  system, 
both  functional  requirements  and 
assurance  requirements,  are  well 
understood  by  the  contractor.  Q4 

C3  The  contractor  understands  how  to 

develop,  operate,  and  maintain  a  secure 
interface  between  classified  processes  and 
unclassified  ones.  Q5  Q6  Q7 


Q1  Describe  your  plans,  including  task 
scheduling,  staffing  allocation,  and 
personnel  training,  for  implementing  the 
security  requirements  of  the  proposed 
development.  Cl 

Q2  Has  a  security  policy  been  defined  for  the 
proposed  system?  Cl 


Q3  Have  you  performed  any  architectural 

studies  to  establish  how  computer  security 
will  impact  the  architecture  of  your 
proposed  development?  Cl 

Q4  Describe  your  understanding  of  the  top- 

level  security  requirements  that  your  system 
must  meet,  including  the  security  modes  of 
operation  that  will  be  used  in  your  system, 
and  the  Trusted  Computer  System 
Evaluation  Criteria  level  that  your  system 
must  meet.  C2 

Q5  Describe  the  design  approach  that  you  will 
use  to  ensure  that  users  and  processes  are 
given  access  to  the  information  and 
processes  they  need  to  perform  their 
assigned  functions  and  that  they  are  not 
given  privileges  beyond  those  required  to 
complete  their  assigned  functions.  C3 

Q6  Describe  the  design  approach  that  you  will 
use  to  minimize  the  amount  of  trusted 
application  code  in  your  system.  C3 

Q7  Describe  the  design  approach  and 

mechanisms  that  you  will  use  to  ensure  that 
classified  and  sensitive  information  is 
protected  within  components  of  your 
system,  at  interfaces  between  system 
components,  and  at  interfaces  between  your 
system  and  external  systems  and  networks. 
C3 
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6  Program  Speciflc  Technologies 

6.5  Trusted  Systems 

6.5.2  Trusted  Systems  Planning 


Cl  The  contractor  understands  the 

certification  and  accreditation  processes 
for  Automated  Information  Systems  that 
are  applicable  to  the  program  (Air  Force, 
Defense  Information  Agency,  National 
Security  Agency,  etc.).  Q1 


Qi  Describe  your  plans  for  ensuring  that  the 
security  accreditation  process  will  be 
successful.  Is  your  plan  based  on  any  past 
experience  with  security  accreditation? 

What  accrediting  agency  did  you  work  with? 
Cl 


C2  The  contractor  has  a  plan  for  supporting 
the  program’s  security  staffing  needs 
consistently  with  the  program’s 
development  schedule.  Q2 


Q2  Describe  your  plans  for  ensuring  that 

people  with  the  required  security  experience 
are  assigned  to  the  program  at  the 
appropriate  development  phases.  C2 
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Program  Specific  Technologies 
Trusted  Systems 

Trusted  Systems  Implementation 

Cl 

The  contractor  understands  how  to 
design  and  implement  a  system  that  meets 
the  Trusted  Computer  System  Evaluation 
Criteria  (TCSEC)  to  the  applicable 

TCSEC  level  (Cl,  C2,B1,  etc.).  Q1  Q2 

Qi 

Describe  your  experience  in  designing  and 
implementing  the  security  mechanisms  that 
meet  the  applicable  TCSEC  requirements. 

Cl 

C2 

The  contractor  understands  how  to  test 
and  verify  high-assurance  hardware  and 
software.  Q3 

Q2 

Describe  your  experience  in  implementing 
assurance  mechanisms  that  meet  the 
applicable  TCSEC  requirements.  Cl 

C3 

The  contractor  understands  how  to 
document  and  operate  a  system  that 
meets  the  TCSEC  to  the  applicable  level 
(C1,C2,  Bl,etc.).  Q4Q5 

Q3 

Describe  your  experience  in  testing  and 
verifying  high-assurance  hardware  and 
software.  How  did  you  budget  resources  to 
support  security  testing  and  verification? 

C2 

Q4 

Describe  your  experience  in  documenting 
your  system  development  to  the  satisfaction 
of  the  applicable  TCSEC  requirements.  C3 

Q5 

Describe  your  experience  in  operating  a 
trusted  system  to  the  satisfaction  of  the 

TCSEC  requirements  applicable  to  the 
program.  C3 

151 


Program  Specific  Technologies 
Distributed  Network-based  Systems  (DNS) 
DNS  Requirements  Analysis 


The  contractor  understands  what  is 
involved  in  the  analysis  and  modeling  of 
DNS  and  has  previous  experience  doing 
analysis  and  modeling  of  DNS.  Q1 

The  contractor  analyzes  how  DNS 
technology  should  be  applied  to  the 
system.  Q2 


The  contractor  models  the  functions, 
behavior,  and  data  of  the  system  and 
compares  alternative  distributed 
architectures.  Q3 

The  contractor  depicts  the  physical 
configuration  of  the  system  and 
compares  it  with  alternative 
configurations.  Q4 

The  contractor’s  analysis  considers  how 
DNS  technology  influences  the  system’s 
performance  of  the  mission.  Q5 


The  contractor  formulates  a 
standardization  policy  for  the  system 
and  identifies  standardization  processes. 

Q6  Q7 

The  contractor  analyzes  how  DNS-type 
requirements  can  contribute  to  the 
system’s  qualities  at  each  phase  of  its 
life  cycle.  Q8  Q9 

The  contractor  evaluates  the  usefulness 
of  various  distributed  services  for  the 
system  and  evaluates  the  applicability  of 
standardized  distributed  services.  QIO 

The  contractor  establishes  distributed 
data  management  requirements  for  all 
data  in  the  system,  and  determines  data 
quality  requirements.  Qll  Q12 


Describe  past  experiences  with  analyzing 
and  modeling  DNSs.  What  were  the 
successes  and  what  were  the  lessons  learned? 

Cl 

Why  is  DNS  technology  chosen  to 
implement  this  system?  What  DNS-related 
needs  such  as  autonomy,  incremental 
expansion,  or  reliability  are  satisfied  by  the 
use  of  DNS  technology?  C2 

Have  you  considered  alternative  system 
architectures  for  functions,  data,  and 
behavior?  Provide  justification  for  the 
selected  ones.  C3 

What  alternative  physical  and  logical 
configurations  were  considered?  Why  was 
the  selected  one  chosen?  C4 


How  does  the  use  of  DNS  technology 
influence  the  system’s  performance  of  the 
mission?  Specifically,  what  are  the  affects 
of  physical  dispersion  of  functions  and  data, 
distributed  synchronization,  distributed 
failures,  and  distributed  security  on  the 
mission?  C5 

What  purposes  are  achieved  by  the 
application  of  standardization  such  as  cost 
reduction,  reusability,  extensibility,  and 
interoperability  to  this  system?  C6 

What  standardization  processes  will  be 
applied  to  the  system?  C6 


How  will  DNS-type  requirements  affect  each 
phase  of  the  system’s  life  cycle,  considering 
that  different  system  components  may 
simultaneously  be  at  different  life-cycle 
stages?  C7 

How  can  the  DNS-type  requirements  used 
for  this  system  influence  the  system’s 
qualities  such  as  reliability,  availability, 
failure  recovery,  flexibility,  and 
extensibility?  C7 
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6. 

6.6 

6.6.1 


Program  Specific  Technologies 
Distributed  Network-based  Systems  (DNS) 
DNS  Requirements  Analysis  (Cont’d) 


CIO  The  contractor  defines  the  DNS-type 
performance  requirements  for  the 
system  and  analyzes  how  those 
requirements  will  evolve  over  the 
lifetime  of  the  system.  Q13  Q14 

Cll  The  contractor  analyzes  the  system’s 
performance  requirements  and 
determines  the  underlying  network  and 
station  platforms  requirements  in  order 
to  support  the  users  and  using 
applications.  Q15  Q16 

Cl 2  The  contractor  identifies  the  users  and 
using  applications  of  the  DNS  and 
analyzes  how  the  use  of  DNS 
technology  supports  the  users  and  using 
applications.  Q17  Q18  Q19 

C13  The  contractor  evaluates  priorities 

among  the  users  and  uses  of  the  system 
to  establish  priorities  among  DNS-type 
requirements.  Q20 


C14  The  contractor  examines  various 

modeling  methods  that  can  be  used  to 
represent  the  architecture  of  the  system, 
and  selects  one  that  is  suitable  for  DNSs. 
Q21  Q22 


QIO  How  does  the  system  make  use  of 

distributed  services,  especially  standardized 
distributed  services  such  as  naming,  time 
synchronization,  network  security,  remote 
operations,  distributed  file  management  and 
data  transport?  Why  are  they  used?  C8 

Qll  How  are  data  qualities  such  as  correctness, 
consistency,  timeliness,  integrity, 
accessibility,  usefulness,  availability,  and 
efficient  storage  incorporated  into  the 
system?  C9 

Q12  What  data  management  strategies  are 

applicable  to  the  system,  considering  the 
need  to  maintain  data  quality  and  the  need 
to  provide  for  concurrent  system  processes? 
C9 

Q13  What  are  the  DNS-type  performance 

requirements  for  the  system  such  as  end-to- 
end  delay,  access  delay,  utilization,  capacity, 
bit  error  rate  (BER),  and  quality  of  service? 
How  do  they  derive  from  the  system 
requirements?  CIO 

Q14  How  are  the  performance  requirements  of 
the  system  expected  to  evolve?  CIO 


Q15  What  are  the  requirements  for  network  and 
station  platforms  such  as  various  types  of 
computer  networks,  multiprocessor 
computers,  or  database  servers?  Cll 

Q16  Why  were  the  selected  set  of  network  and 

station  platform  requirements  chosen?  Cll 

Q17  Who  are  the  users  and  what  are  the  using 
applications  of  the  DNS?  C12 

Q18  How  well  are  the  using  applications  and 
personnel  roles  supported  by  the  use  of 
DNS  technology  for  this  system?  Cl 2 

Q19  What  is  the  usefulness  of  applying  process 
reengineering,  applied  to  the  human- 
computer  processes  of  the  system,  for  the 
DNS-type  requirements  of  the  system?  C12 
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6. 

6.6 

6.6.1 


Program  Specific  Technologies 
Distributed  Network-based  Systems  (DNS) 
DNS  Requirements  Analysis  (Cont’d) 


Q20  What  are  the  priorities  and  where  should 
then  be  applied  to  the  functions,  data,  and 
behaviors  of  this  system?  C13 

Q21  What  alternative  modeling  methods  were 
considered  for  the  DNS?  Why  was  the 
selected  one  chosen?  C14 

Q22  What  are  the  advantages  and  disadvantages 
of  using  object-oriented  modeling  (OOM)? 
Which  OOM  methodology  would  be  best? 

C14 
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6. 

6.6 

6.6.2 


Program  Specific  Technologies 
Distributed  Network-based  Systems  (DNS) 
DNS  Design  Technology 


Cl  The  contractor  understands  DNS 

technology  and  has  previous  experience 
with  design  of  DNSs.  Q1  Q2 

C2  The  contractor  understands  and  has 

experience  with  open  systems  concepts 
such  as  adoption  of  industry  standards. 

Q3  Q4  Q5 

C3  The  contractor  identifies  computer 

networking  and  station  technologies  and 
requirements  that  form  a  DNS  platform 
for  the  system  and  has  an 
implementation  approach  for  the  system 
using  the  selected  technologies.  Q6  Q7 
Q8 

C4  The  contractor  identifies  applicable 

computer  application  architectures  that 
meet  the  DNS  platform  requirements  for 
the  system  and  an  initial  architecture. 

Q9  QIO 

C5  The  contractor  understands  automated 
distributed  systems  management  and 
designates  applicable  features  to  the 
system  design.  Qll  Q12 

C6  The  contractor’s  design  incorporates 
DNS  maintenance  and  support 
functions.  Q13  Q14 

C7  The  contractor  ensures  achievement  of 
the  system’s  qualities  and  presents  an 
approach  for  achieving  data  quality 
requirements.  Q15 

C8  The  contractor  ensures  the  appropriate 
amount  of  access  transparency  and  uses 
appropriate  DNS  services  for  the  system. 

Q16  Q17 

C9  The  contractor  provides  for  the 

appropriate  amount  of  security  and 
considers  security  in  the  use  of  DNS 
services  where  applicable.  Q18 

CIO  The  contractor  understands  and 

incorporates  DNS-specific  characteristics 
applicable  to  the  system.  Q19  Q20 


Qi  Describe  past  experiences  with  designing 
DNSs.  What  were  the  successes  and  lessons 
learned?  Cl 

Q2  Is  heterogeneity  in  applications  types, 

vendors,  hardware  platforms,  and  computer 
network  protocols  applicable  to  this  system? 

Cl 

Q3  Describe  your  experience  with 

implementing  open  systems.  What  have 
been  the  successes  and  lessons  learned?  C2 


Q4  What  specific  standards  and  types  of 

standards  are  applicable  to  the  system  under 
development?  C2 


Q5  How  will  standards  be  considered, 

promulgated,  adopted,  created,  and  updated 
during  development  and  support  of  the 
system?  C2 

Q6  What  types  of  computer  networks  and 

station  technologies  are  applicable  to  this 
system?  Why  are  they  applicable?  C3 

Q7  What  types  of  computer  network 

requirements  are  needed  by  this  system? 

C3 


Q8  What  is  the  implementation  approach  for  the 
platform  computer  network  and  stations? 

C3 


Q9  What  types  of  application  architectures  such 
as  varieties  of  client-server,  are  applicable  to 
the  system?  Why  are  they  applicable?  C4 

QIO  What  is  the  best  way  to  implement  the 

selected  application  architecture  to  meet  the 
requirements  for  the  system?  C4 
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6.  Program  Specific  Technologies 

6.6  Distributed  Network-based  Systems  (DNS) 

6.6.2  DNS  Design  Technology  (Cont’d) 


Cll  The  contractor  has  an  approach  for 

measuring  and  anticipating  the  need  for 
expanded  system  capacity.  Q21 

C12  The  contractor  provides  for  automated 
measurement  and  optimization  of  the 
DNS  services  to  meet  the  system’s 
performance  requirements.  Q22  Q23 
Q24  Q25 


Qll  What  are  alternative  system  management 
approaches?  Why  is  distributed  system 
management  needed  or  not  needed  for  the 
system?  C5 

Q12  What  provisions  are  made  for  the  following 
system  management  areas:  performance 
management,  fault  management,  account 
management,  security  management,  network 
configuration  control  (note:  this  is  control 
of  operations,  not  design),  and  network 
planning.  C5 


Q13  Does  the  development  approach  integrate 
the  development  functions  of  the  system 
with  its  support  functions  after  it  becomes 
operational?  C6 

Q14  Does  the  development  approach  unify  the 
administration  of  the  system  for  all  life- 
cycle  phases?  C6 

Q15  How  will  the  adopted  computer  network 
technologies,  application  architectures, 
distributed  systems  management,  backup, 
archiving,  and  recovery  methods  achieve 
quality  requirements,  and  (particularly)  data 
quality  requirements  of  the  system?  C7 

Q16  How  will  DNS  services  in  general  and 

access-related  services  in  particular  such  as 
naming,  addressing,  routing,  and  directory 
services  be  used  in  the  system?  C8 

Q17  What  amount  of  access  transparency  is 
needed  for  the  system?  How  will  it  be 
implemented?  C8 

Q18  What  degree  of  security  is  required  for  the 
system?  How  will  it  be  implemented?  C9 

Q19  How  will  the  DNS  services  be  used  to 

implement  security,  considering  the  special 
security  problems  of  DNSs?  CIO 

Q20  How  will  DNS-specific  characteristics  such 
as  distributed  file  management,  remote 
interprocess  communications,  load 
balancing,  copy  management,  distributed 
resource  scheduling,  and  distributed  timing 
synchronization  be  used?  CIO 

Q21  How  will  upgrades  or  insertion  of  new 

technology  be  handled  when  there  is  a  need 
to  respond  to  increasing  demands?  Cll 
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6.  Program  Specific  Technologies 

6.6  Distributed  Network-based  Systems  (DNS) 

6.6.2  DNS  Design  Technology  (Cont’d) 

Q22  Will  testing,  monitoring,  simulation,  or 
emulation  of  the  distributed  services  be 
required  for  the  DNS  system?  C12 

Q23  What  are  appropriate  goals  for  optimization 
of  the  DNS  services?  C12 

Q24  How  will  DNS-related  performance  be 
measured  and  assessed?  C12 

Q25  What  measurements  are  needed  to  optimize 
the  system?  C12 
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6.  Program  Specific  Technologies 

6.6  Distributed  Network-based  Systems  (DNS) 

6.6.3  DNS  Development  and  Support  Tools 


Cl  The  contractor  understands  tools  suitable 
for  DNSs,  and  has  experience  using  tools 
on  DNSs.  Q1 

C2  The  proposed  system  has  designed-in 
features  to  allow  development  and 
support  (such  as  administration)  tools  to 
adequately  monitor  and  control  the 
system.  Q2 

C3  The  contractor  proposes  a  set  of  tools 
that  are  suitable  for  developing  and 
supporting  DNSs.  Q3 


C4  The  contractor  proposes  an  integrated 
suite  of  development  and  support  tools 
that  automatically  exchange  information 
using  standards.  Q4 

C5  The  contractor’s  internal  office 

automation  system  will  be  integrated  with 
the  development  and  support  tools.  Q5 


C6  The  contractor  proposes  a  set  of  tools  to 
help  optimize  the  DNS-type  performance 
characteristics  of  the  system.  Q6 


Ql  Describe  your  experience  using  DNS  tools 
for  development  and  support  of  DNSs. 

What  were  the  successes  and  lessons 
learned?  Cl 

Q2  What  design  features  should  the  system  have 
that  help  development  and  support  tools 
perform  their  functions?  C2 


Q3  What  tools  do  you  own  or  plan  to  acquire 
for  this  system  that  provide  for  DNS 
partitioning  of  applications,  including  both 
contractor-  and  Government-funded  ones? 
C3 

Q4  In  what  ways  are  the  proposed  development 
and  support  tools  integrated?  For  instance, 
do  they  exchange  information  using 
standards  such  as  Simple  Network 
Management  Protocol  (SNMP)?  C4 

Q5  How  does  your  office  automation  system 
integrate  with  contractor-owned  and 
-planned  development  and  support  tools 
including  both  contractor-  and 
Government-funded  ones?  C5 

Q6  How  do  the  proposed  tools  perform  the 
measurements  needed  to  optimize  the 
system?  How  do  the  tools  help  achieve 
optimum  performance?  C6 
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Program  Specific  Technologies 
Distributed  Network-based  Systems  (DNS) 
DNS  Engineering  Management  Methods 

Cl 

The  contractor  establishes  procedures  to 
reuse  or  integrate  legacy  components, 
and  establishes  procedures  for  migrating 
applications  from  legacy  platforms  to 
new  ones.  Q1  Q2 

Q1 

What  procedures  are  established  to  reuse 
legacy  components  and  migrate  applications 
from  legacy  systems  to  new  ones?  Cl 

C2 

The  contractor  institutes  procedures  to 
control  the  cost  of  the  system  and  to 
adjust  expenditures  based  on  updated 
system  development  priorities.  Q3  Q4 

Q2 

What  methods  are  used  to  integrate  legacy 
and  newly  developed  portions  of  the 
system?  Cl 

C3 

The  system  design  approach  allows 
various  components  to  be  simultaneously 
at  different  life-cycle  phases.  Q5 

Q3 

What  procedures  are  instituted  to  control 
costs  and  to  adjust  expenditures  based  on 
development  system  priorities  which  may 
change?  C2 

C4 

The  contractor  has  an  approach  for 
inserting  new  technologies  in  the  DNS. 

Q6 

Q4 

How  is  a  cost-benefit  analysis  used  to  guide 
the  development  and  support  of  the  system? 
C2 

Q5 

What  methods  will  be  used  to  engineer  and 
manage  different  components  of  the  system 
that  are  simultaneously  at  different  life- 
cycle  phases?  C3 

Q6 

What  methods  are  used  to  ease  technology 
insertion  for  the  DNS  characteristics  of  the 
system?  C4 
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6.  Program  Specific  Technologies 

6.7  Object  Oriented  (00)  Developments 

6.7.1  Development  Process  and  Methodologies 


Cl  The  contractor  has  selected  a 

development  process  that  will  minimize 
the  major  process  risks  associated  with 
00  developments.  Q1  Q2  Q3 


C2  The  contractor  has  demonstrated  the 
ability  to  control  the  quality  of  an  00 
design.  Q4  Q5  Q6  Q7 

C3  The  contractor  has  demonstrated  the 

ability  to  combine  00  systems  with  non- 
00  systems  in  its  development.  Q8 


Qi  Describe  your  overall  development  process, 
how  the  different  development  activities  are 
sequenced,  the  expected  outputs  from  each 
activity,  what  percentage  of  the  schedule  is 
allocated  to  each  activity,  what  criteria  are 
used  for  proceeding  from  one  activity  to  the 
next  and  who  controls  that  decision.  Cl 

Q2  Describe  the  metrics  you  use  for  tracking 
and  monitoring  your  development  cost  and 
schedule,  how  often  they  are  measured  and 
analyzed,  and  by  whom.  Cl 

Q3  Describe  your  process  for  controlling  and 
coordinating  requirements  and  design 
changes.  Cl 

Q4  Describe  your  process  for  defining  objects, 
classes,  and  class  hierarchies,  and  the 
relationships  between  them.  C2 

Q5  Describe  your  process  for  defining  and 
documenting  the  high-level  system 
architecture  and  the  static  and  dynamic 
models  that  will  be  employed  in  the  analysis 
and  design  of  the  system.  C2 

Q6  Describe  your  process  for  evaluating  the 
quality  of  your  design  and  for  controlling 
its  complexity,  and  the  metrics  used  to  that 
effect.  C2 

Q7  Describe  your  process  for  ensuring  that  the 
soundness  of  the  overall  architecture  is 
maintained.  C2 

Q8  Describe  any  experience  you  may  have  in 
interfacing  an  OO  system  with  legacy  or 
commercial  off-the-shelf  software  that  is  not 
00.  For  each  example,  describe  how  you 
interfaced  the  design  methodologies  and  the 
development  languages  associated  with  each 
system.  C3 
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Program  Specific  Technologies 
Object  Oriented  (OO)  Developments 
Tools,  Language  and  Related  Technologies 


The  contractor  has  selected  development 
tools  that  are  appropriate  for  the 
program.  Q1  Q2  Q3  Q4 


The  contractor  has  a  good  understanding  Q2 
of  the  development  language  selected  for 
the  program.  Q5  Q6 

The  contractor  has  adequate  experience  Q3 
with  OO  technologies  applicable  to  the 
program  Q7 


Describe  in  tabular  form  the  development 
tools  that  will  be  used  on  the  program,  what 
development  activity  they  support,  what 
methodology  they  support,  what  their  origin 
is  (Government-furnished  equipment, 
vendor-supplied,  in-house  developments, 
etc.)  and  what  plans  you  have  for 
maintaining  them.  Cl 

Assess  the  maturity  of  each  of  the  selected 
OO  tools,  and  explain  the  basis  for  your 
assessment.  Cl 

Describe  any  guidelines  you  have  for  the 
use  of  your  OO  tools.  Describe  any  areas 
where  the  selected  tools  have  weaknesses 
and  describe  your  plans  to  mitigate  them. 

Cl 

Describe  any  studies  you  may  have  on  how 
well  the  tools  will  scale  up  to  the 
development  at  hand  in  terms  of  number  of 
users,  performance,  and  memory  use.  Cl 

Describe  the  programming  language 
proposed  for  the  program,  and  your 
experience  with  the  language  and  the 
associated  compile  and  build  tools.  C2 

Describe  any  plans  and  associated  rationale 
you  may  have  for  restricting  the  use  of 
specific  features  of  the  language.  Describe 
any  OO  constructs  (e.g.,  multiple 
inheritence)  you  plan  to  use  that  are  not 
directly  supported  by  the  language,  and 
describe  your  plan  to  work  around  these 
deficiencies.  C2 

Describe  your  experience  with  00-related 
technologies  such  as  object  request  brokers, 
object-oriented  database  management 
systems,  etc.  C3 
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6.  Program  Specific  Technologies 

6.7  Object  Oriented  (00)  Developments 

6.7.3  Performance 

Cl  The  contractor  has  demonstrated  the 
ability  to  control  the  timing  overhead 
associated  with  OO  developments.  Q1 

Q2Q3 

Q1 

Describe  your  efforts  at  minimizing  the 
overhead  associated  with  layering.  Cl 

C2  The  contractor  has  demonstrated  the 
ability  to  mitigate  memory  management 
risks  associated  with  OO  developments. 

Q3  Q4  Q4  Q5  Q6 

Q2 

Describe  your  efforts  to  analyze  the 
overhead  associated  with  message  passing. 
Have  you  done  an  analysis  of  the  messaging 
overhead  expected  on  the  program  at  hand? 
Cl 

Q3 

Describe  your  use  of  inlining  functions  and 
the  associated  price  in  memory  use.  Cl  C2 

Q4 

Describe  your  efforts  to  plan  and  manage 
your  memory  resources  effectively.  C2 

Q5 

Describe  your  efforts  to  analyze  the  use  of 
dynamic  memory  allocation.  Do  you  have 
any  plans  for  when  and  how  to  use  dynamic 
memory  allocation?  What  risk  mitigation 
steps  will  you  take  if  no  memory  is  available 
when  the  allocation  is  done?  C2 

Q6 

Explain  how  garbage  collection  will  be 
handled.  C2 
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I.  Program  Specific  Technologies 

.7  Object  Oriented  (OO)  Developments 

.7.4  Benefits  Achieved 


Cl  The  contractor  has  demonstrated  that  its 
use  of  OO  methods  will  yield  reuse 
benefits.  Q1  Q2  Q3  Q4  Q5 

Ql 

Describe  the  class  libraries  you  have,  what 
their  origins  are  (in-house  development, 
vendor-supplied,  public  domain,  etc.),  and 
what  experience  you  have  with  them.  Cl 

C2  The  contractor  has  demonstrated  that  its 
use  of  OO  methods  will  yield 
maintainability  benefits.  Q6  Q7  Q8 

Q2 

Describe  the  domains  your  class  libraries 
cover,  and  explain  their  applicability  to  the 
program  at  hand.  Cl 

Q3 

Describe  your  process  for  developing  and 
maintaining  class  libraries,  the  acceptance 
criteria  used  for  configuring  them,  and  how 
often  they  are  changing.  Cl 

Q4 

If  multiple  contractors  are  involved,  describe 
your  process  for  identifying  and  sharing 
reusable  products  between  contractors.  Cl 

Q5 

Describe  the  percentage  of  reuse  you 
achieved  on  past  programs  and  explain  how 
that  percentage  is  derived.  Cl 

Q6 

Describe  any  trade-off  studies  you  have 
regarding  maintainability  benefits  resulting 
from  your  OO  developments.  C2 

Q7 

Describe  your  experience  with  adding 
features  to  an  OO  system,  and  explain  how 
the  cost  compares  with  a  situation  where  the 
system  was  not  OO.  C2 

Q8 

Describe  any  exprience  you  have  with 
another  organization  maintaining  OO 
products  developed  by  your  organization. 

C2 
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6.  Program  Specific  Technologies 

6.7  Object  Oriented  (OO)  Developments 

6.7.5  Organizational  Commitment  and  Experience 


Cl 

The  contractor  has  extensive  experience 
with  OO  developments.  Q1  Q2  Q3  Q4 

Qi 

Describe  the  other  OO  developments  you 
are  or  have  been  involved  in;  their  missions, 
phases  (demonstration/validation,  full  scale 
development,  etc.),  and  current  status.  Cl 

C2 

The  contractor  has  a  good  OO  training 
program.  Q5  Q6 

Q2 

Describe  your  experience  with  managing 
and  developing  OO  systems.  What 
percentage  of  your  team  is  OO  experienced 
and  what  is  the  level  of  experience  in  terms 
of  number  and  size  of  programs?  Cl 

C3 

The  contractor’s  decision  to  pursue  an 

OO  development  is  backed  by  a  strong 
resource  commitment.  Q7  Q8 

Q3 

Describe  the  level  of  experience  of  your 
most  senior  OO  experts  and  their 
commitment  to  this  program.  Describe 
your  plans  for  deploying  them.  Cl 

Q4 

Describe  the  pool  of  OO  experts  you  can 
draw  from  for  the  program  at  hand;  their 
experience  and  current  commitments. 
Describe  your  plans  for  deploying  them. 

Cl 

Q5 

Describe  the  training  available  in  the  use  of 
the  selected  OO  methods  and  tools,  the 
source  of  the  training,  and  the  steps  taken 
for  encouraging  your  personnel  to  obtain 
adequate  training.  C2 

Q6 

Describe  your  efforts  for  keeping  your 
personnel  current  in  OO  technologies.  C2 

Q7 

Explain  how  you  will  ensure  that  the 
number  of  software  development  platforms 
and  tool  licenses  will  remain  adequate  for 
the  duration  of  the  program.  C3 

Q8 

Explain  how  you  will  ensure  that  the 
developers  have  adequate  support  in  their 
use  of  the  selected  tools  and  methodologies 
for  the  duration  of  the  program.  C3 

164 


References 


1 .  “Acquisition  -  Software  Development  Capability  Evaluation,”  Air  Force  Materiel 
Command,  AFMC  Pamphlet  63-103,  Volumes  1  and  2(15  June  1994). 

2.  “Acquisition  Management,  Software  Development  Capability/Capacity  Review,” 
Aeronautical  Systems  Center,  ASC  Pamphlet  800-5  (11  September  1992). 

3.  Paulk,  M.,  B.  Curtis,  M.  Chrissis,  and  C.  Weber,  “Capability  Maturity  Model  for  Software 
(version  1.1),”  CMU/SEI-93-TR-024  (1993). 

4.  Babel,  P.,  R.  Haddad,  and  E.  Shulman,  “Software  Development  Capability  Evaluation, 
Experiences  and  Lessons  Learned,”  Software  Technology  Conference  (May  1997). 

5.  Haddad,  R.,  M.  Lubofsky,  and  B.  White,  “Pre- Award  Survey  Method  for  Software 
Acquisition  at  Space  and  Missile  Systems  Center,”  TOR-92(2904)-4,  The  Aerospace 
Corporation  (18  September  1992). 

6.  Haddad,  R.  “New  Criteria  For  the  Software  Development  Capability  Evaluation,”  TOR- 
95(5917)-3,  The  Aerospace  Corporation  (September  1995). 

7.  Haddad,  R.  “Guidelines  for  the  Use  of  SDCE,”  TR-98(8550)-2,  The  Aerospace  Corporation 
(March  1998). 


165 


